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"[...] it is only on condition that he departs, in a second rupture, the illusion of the neutrality and 
independence of the school system in relation to the structure of class relations that one can 
come to question the question about the exam to find out what the hidden exam and what the 
question about the exam contributes to hide by deviatingthe question of elimination without 
examination" Pierre Bourdieu and Jean Claude Passeron (1982). 


"If we consider that the quality of school education is not only based on its scientific and 
pedagogical quality, but also, at the same time, on its democratic quality, We cannot fail to 
demand that the evaluation of public schools be structured through more complex procedures, 
diversified and very different from those that have been based on standardized or standardized 
external examinations" Almerindo Janela Afonso (2001). 


"Education is a social process, it is development. It is not the preparation for life, it is life itself. 
We learn when we share experiences" John Dewey (1982). 


"The main goal of education is to create men who are able to do new things, not simply repeat 
what other generations have done. Men who are creators, inventors, discoverers. The second 
goal of education is to form minds that are in a position to criticize, verify and not accept 
everything that is proposed to them" Jean William Fritz Piaget (1978). 


"[...] Think practice as the best way to perfect practice. Think of the practice through which it 
recognizes the theory embedded in it. The evaluation of practice as a path of theoretical 
formation and not as a mere recrimination instrument of the teacher" Paulo Freire (2001). 


"Education is the point where we decide whether we love the world enough to take responsibility 
for it. Loving life is easy when you’re abroad. Where no one knows you and it is you who hold 
your life in your own hands, you are master of yourself more than at any other time" Hannah 
Arendt (1970). 


"Assement as we know it Did not Simply appear one day; it is the result of a developmental 
process of Construction and reconstruction that involves a number of interacting influences" 
Free Translation. "The evaluation as we know it did not come in just one day; this is the result 
of a process of development, construction and reconstruction that involves various influences 
in interaction" Guba & Lincoln (1989, p. 22). 


A new report by Fairtest - USA (National Center for Fair and Open Tests), declares that the 
Educational Assessment is Building Model State Systems to Replace the Exaggeration of Tests, 
further accuses the "test and punish approach" incorporated into federal legislation. Producing 
"mass layoffs, school closures and deterioration of the educational climate that fed the school’s 
pipeline to prison" Alan Singer (2020). 


The act of evaluating should be a constancy, as it should be understood as the construction of 
the self-portrait. It is necessary to examine the practices, to know what it is and, above all, where 
it is intended to arrive (Vilas-Boas, 2012). 
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ABSTRACT 


VIEIRA NETO, Leopoldino. Modeling in Education: The Brazilian System of 
Teaching, Evaluation, Metrics and Parameters in Learning. Florida, 2020, 130fls. 
Thesis ( Doctorate in Educational Sciences ) Graduate Program in Educational 
Sciences at Florida University - USA, 2020. 


This thesis focused on the changes in the Educational System by students’ 
proficiency results in PISA testing. The general objective was to highlight the 
evaluative practices in the researched education systems, confronted Brazil, USA 
and Portugal; The multiple analyses involving the quality of education were also 
described. To carry out this study, a qualitative-quantitative, descriptive research 
was Carried out in the form of data collection in bibliographic research and in the 
OECD/PISA reports, practices reiterated in the documents/objects listed for this 
study. The variables: Schooling, Teachers, Performance, Enrollment, 
Investments and Student-teacher Ratio were analyzed in light of the results. The 
results are presented as a proposal for the construction of the educational system 
as a measure for increase in the OECD ranking. 


Keywords: [Evaluation - Educational System - Educational Policies - 
International Student Assessment Program (PISA). 


RESUME 


VIEIRA NETO, Leopoldino. Modélisation dans l'éducation: le systeme brésilien 
d'enseignement, d'évaluation, de métriques et de parametres d'apprentissage. Floride, 
2020, 130fls. These ( doctorat en sciences de l'éducation ) Programme d'études 
supérieures en sciences de l'éducation a l'Université de Floride - Etats-Unis, 2020. 


Cette these portait sur les changements dans le systeme éducatif en raison des 
résultats de competence des étudiants aux tests PISA. Comme _ objectif général, nous 
avons cherché a mettre en évidence les pratiques évaluatives dans les systemes 
éducatifs confrontés au Brésil, aux USA et au Portugal, les multiples analyses impliquant 
la qualité de l'éducation ont également éte décrites. Pour mener a bien cette étude, une 
recherche qualitative et quantitative a été menée, sous forme de collecte de données 
dans la recherche bibliographique et dans les rapports OCDE /PISA, pratiques répétees 
dans les documents / objets répertoriés pour cette étude. Les variables: scolarité, 
enseignants, performances, inscriptions, investissements et ratio éleves-enseignant ont 
ete analysées a la lumiére des résultats. Les résultats sont présentés comme une 


proposition visant a construire le systeme éducatif comme une mesure pour augmenter le 
classement de l'OCDE. 


Mots clés: Evaluation - Systeme éducatif - Politiques éducatives - Programme international 
d'évaluation des éleves (PISA). 
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1 INTRODUCTION 
1.1 THEME 


Education is synonymous with transformation. And a universal social 
phenomenon that seeks in its framework the promotion of human development from 
its Genesis to adulthood. The human being is educated by the set of experiences 


felt, lived and experienced. 


Among the most primitive peoples, education was differentiated from 
what we have today, we find education through imitation practices, reflecting moral, 
political, social and religious values. Currently, education is proposed by a learning 
teaching process where the student suffers endogenous (mind) and exogenous 


(environmental) stimuli that he receives as educational practice. 


In order to obtain an adequate perspective of modeling as a central skill 
in scientific reasoning, providing a framework for educational knowledge, we 
present this study that elaborates, tests and reviews models as a crucial aspect of 
understanding education science. However, modeling formalisms have traditionally 
been associated with the creation of mathematical models for obtaining numerical 


results. 


Such approaches fail to capture many crucial aspects of the models, 
such as the conditions under which a model is applicable, and are relatively 
inaccessible to teaching, for students. On the other hand, qualitative formalisms of 
reasoning present only a conceptual analysis of the system including only the cause 
not demonstrating the result. 


Given that qualitative modeling of processes’ is a valuable technology 
for education, we point out that most educators are only concerned with socializing 
conceptual knowledge. Not caring in the praxis with the teaching process learning 
as a parameter of measuring the quality of Teaching.? 





1 The term: qualitative process modeling can be defined as: Qualitative Process Theory, several different 
types of reasoning that can be performed with them and discusses their impact on other issues in common 
sense reasoning about the physical world, such as causal reasoning and interpretation measuring Forbus, 
K. (1984). http://hdl.handle.net/1721.1/6874 





2 Quality measurement is the perception of the service received comparing it with what they consider 
appropriate, and between them there is a zone of tolerance that is based on the characteristic variability of 
services in the Model Service Quality Gap Analysis (GRONROOS, 1984). 
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However, discovering how we think about learning entities and 
processes is one of the central scientific goals of qualitative education. Qualitative 
progress has led to new modeling languages that describe entities and processes 
in conceptual terms, incorporate natural notions of causality and express 
knowledge about the modeling process itself (FALKEN-HAINER AND FORBUS 
[1991]; FORBUS [1996]; RICKEL AND PORTER [1997]; WELD E DE KLEER 
[1990]). 

These "qualitative and quantitative" semantic fields provide new 
resources for teaching-learning. By incorporating metric models of® entities and 
processes in education, the understandings of these parameters can be used to 
provide explanations directly to the education system, showing how the specific 
results were obtained. 

Recently, qualitative and quantitative environments of model building 
have been proposed that allow students to evaluate knowledge articulating using 
graphic representations of these intuitive notions PISA (Australian Council for 
Educational Research (ACER) Netherlands National Institute for Educational 
Measurement (CITO) Educational Testing Service (ETS, USA) National Institute for 
Educational Policy Research (NIER, Japan) Westat (United States) Bessa 
Machado and Bredeweg, 2002, 2001; Biswas et al. 2001; Forbus et al. 2001. In this 
thesis, we will explore more such approaches. 


The studies found on Educational Systems structures show 
broad approaches and multiple analyses and deepening. 
The complexity and breadth of this theme are one of the 
factors that discourage researchers from writing and 
reporting on these complex organizations. The difficulty of 
a singular and comprehensive look that explains the whole 
system - although limited to the specificities of analysis of a 
given. 


the system - even if limited to the specifics of analysis of a given 
question - requires a deep knowledge of the system and the functioning of 
the object studied. 


Based on these’ premises’ evidenced by: _ difficulties, 
comprehensiveness, complexity and multiple environments of the theme studied, 
the use of Actors evidenced by Chanlat (1995) that describes the influence and 
power of social actors, identifying and analyzing the strategies used qualitative 
or quantitative as a way to achieve the objectives. 


3 Metric Models understood as: Statistical techniques that focus and present the structure of simultaneous 
relationships between 3 or more phenomena. Multivariate Data Analysis (Hair et al., 2009). 
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1.1 JUSTIFICATIVE 


We hope to contribute in this study with teaching-learning a 
worldwide, current and dynamic phenomenon that is qualitative and quantitative 
evaluation of educational systems, evidenced by the Programme for International 
Student Assessment. The OECD. This metric has the power to establish a rank 
among countries, measuring and qualifying educational systems, a factor entirely 
controllable from a strategic point of view, it is important to know the dynamics of 
education as a social factor to, at least, monitor and formulate more appropriate 
decisions. Moreover, we seek to know these phenomena: PISA, PIRLS, TIMSS, 
eTIMSS, ICILS and ESLC that affect, for example, teaching learning processes 
around the world, as a tool of a competitive market, but still an aspect to which 
little attention has been given compared to functional issues of the systems. 


Working in education for thirty years, and occupying a position as 
Teacher, Director and Counselor in Educational Management Bodies such as: 
CEE - Conselho Estadual de Educagao do Espirito Santo (State Council of 
Education of Espirito Santo) and CTEDEMEC - Conselho Técnico Educacional 
(Technical Educational Council) in the Police Station of the Ministry of Education 
and finally elected President of CREP/ES - Conselho Regional de Educadores e 
Pedagogos do Espirito Santo (Regional Council of Educators and Pedagogues 
of Espirito Santo). | am thus qualified to discuss and analyze the "Evaluation 
State" and its structures and Educational Systems. 


As a Mathematics Licensed Professor we have had Experiences in: 
Higher Education (Federal Education System), in Technical Education (State 
System), in the (Municipal Education System) and finally, as a Private School 
Teacher (Private Education System). Thus, | know the four teaching structures 
and their evaluations regarding quality. 


1.2 GENERAL OBJECTIVE 


The general objective of this thesis is to identify the educational practices 
evidenced as Evaluations in Teaching Systems, describing the processes 
involved in these actions, that is, why the related Actors build their own 
environments, dissociated from the guidelines established nationally, causing 


a difference in learning and being reflected in evaluations. 
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1.2.1 Specific Objectives 


Identify consolidated practices in teaching-learning processes; 
Analyze the results measured by the National Evaluation Systems; 


Describe the multiple analyses involved in the quality of Education; 


MMM M 


Relate the teaching-learning models in the studied Systems. 


1.3 THE RESEARCH PROBLEM 


The result of Pisa creates a phenomenon called accountability a mechanism 
of the "Evaluating State", this phenomenon allows each country to evaluate 
the knowledge and skills of its students compared to those of other countries, 
learning from the policies and practices applied elsewhere, as well as 
formulate their educational policies and programs, aiming at improving the 
quality and equity of learning outcomes. Based on these premises, can we 
qualify that Brazil is going through a situation of uncontrolled educational 


practices and loss of quality in Education? 


1.4 WORK DELIMITATION 


The general proposition, defined here as the delimitation of the work, are the 
practices defined in educational programs and Teaching Systems will be 
studied based on Assessments. As a decision of time, in the period from 2000 
to 2020 (twenty) years will allow a longitudinal study and determination of 


trends of the actors involved in the Systems. 





4 The word first appears in Anglo-Saxon dictionaries in the 18th century. It was used in the 1970s (20th 
century), which became a strong manifestation in the educational field. It means: to render accounts 
(accountability) with the possibility of sanction. Boves, 2007; Mulgan, 2000; Schedler, 2004. 
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1.5 STRUCTURE 


This thesis presents in Chapter 1, introduction, justification, objectives, 
research problem, work delimitation and research methodology. Chapter 2, a 
bibliographical review on Evaluation, focusing on concepts and history on the 
theme, Systems Involved and Social Structures. Chapter 3, The Evaluation State, 
its measures and parameters. Chapter 4 addresses the History of Evaluation in 
Brazil, its structure, models, types of evaluation and measurement of results. 
Chapter 5 highlights the evaluation in the European Community and its relevant 
results. Chapter 6 discusses the Evaluation in the United States and its matrix for 
modeling outcomes in Educational Evaluations, Chapter 7 describes the 
methodology used in this thesis, focusing the methodological strategy to the 
nature of the study and the method of research, Chapter 8 presents the models 
and relevant results. Chapter 9 presents and defines the practices to be 
considered as a model. And, finally, Chapter 10, the final considerations are 
presented, containing the contributions and limits of the research, suggestions 
and answers for the development of other studies related to the theme treated 
here. 


1.6 RESEARCH METHODOLOGY 
According to Duarte and Furtado (2002, p.15) research is "a rational 
and systematic procedure that aims to provide answers to the problems 
curiosities or challenges that are proposed. The methodological procedures 
necessary to carry out this research are part of the qualitative and quantitative 
approach, allowing to describe, analyze and effectively understand the 
modeling process of the Systems involved in the study using triangulation of 


data. 

The option for the quali-quantitative® approach is_ valid, 
understanding that the phenomena described in this study will be explained 
by the approaches due to the degree of complexity and nature of the problem 
studied. In addition, the option proves to be the most appropriate to the 
problem and the intended analysis model, and for having as its objective the 


identification of information that is measurable and qualified in this research. 





> Creswell and Clark (2007), present the term as a typology focused on the social sciences. The quantitative 
approach, complementing the qualitative one. 
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2 THE GENESIS OF ASSESSMENT 


In this chapter we will describe the historical evidence that reports the 
development of Educational Assessment and the evolution of evaluative 


practices. 


2.1 BRIEF REPORT ON ASSESSMENT THROUGHOUT HISTORY 


Assessment is not a contemporary practice; on the contrary, it has 
been carried out for a long time in the world since the primitive peoples. 
Millennia ago, Chinese and Greeks already created criteria to select 
individuals to assume certain works (DIAS, 2002) quoted by Rossato, 2004. 
According to Santos (2014, p. 3) some peoples were already using this 


practice: 


In China, in 360 B.C., due to this system, all citizens have 
the possibility to reach prestigious and powerful 
positions. In Greece, Socrates suggested self-evaluation 


- knowing yourself as a requirement to reach the truth. 
Another form of evaluation was carried out through oral 
exercises used by medieval universities and later by 
Jesuits. 


Understanding that Evaluation was an instrument of selecting power, 
these people used it to characterize qualified labor, thus determining who 


occupied certain prestigious positions. Still according to Santos (2014, p. 3) 


emphasizes: 


In the Middle Ages, universities had as their main 
objective the training of teachers. The students who 
completed the bachelor’s degree needed to pass an 
exam in order to teach, and the masters only received 
doctoral degrees if they publicly read Pedro Lobardo’s 
book or later defended theses (SOEIRO & AVELINE, 
1892) quoted by Rossato, (2004). 


Thus, the evaluation practice as a selection process restricts the 
access of the mass to positions, positions and functions separating each 
individual from the others. Correia (2018) states that the known form of the 
school examination as we know it today, was conceived throughout the 16th 
century until the middle of the 17th century with the document "Ratio atque 
Institutio Studiorum Societatis ] esu" Sin a free translation. It was also called 
the Ratio Studiorium. 





® Ratio Studiorum, the method remained for almost two centuries, until the suppression of the order in 
1773, when Pope Clement XIV prohibited the Society of Jesus from acting in its colleges. Franca (1952). 
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According to Stufflebeam and Shinkfield (1993), two movements 
emerged at the end of the 19th century, considered pre-style, the first in the 
United States of America and the other in England. In the USA the emphasis 
on the use of tests to evaluate school performance in Boston school, promoted 
by Harace Mann’ . The English, on the other hand, implement the "real 
commissions", aiming to evaluate public services. 


The same authors highlight the development by Josep M. Rice 
(apud STUFFLEBEAM; SHINKFIELD, 1993) of an evaluation model between 


1887 and 1898 that analyzed 33,000 (thirty-three thousand) students, being 
verified in learning spelling, also cited by Guba and Lincoln (1989). 


The Evaluation has very old roots according to Barbier (1985); Guba 
and Lincoln (1989); Stufflebeam and Shinkfield (1993); the first registered 
movement of this origin in the United States was in The Common J ournal, 


64 of 1845 having as author Mann (1845), was argued the test of "printed 
questions and written answers", considered by evaluators as "novel mode of 
examination" in America thus formatting a new history in North American 
education. 


Emphasizing the evaluation Mann (1845, p. 330) describes that, "But 
when the questions are the same, there is equality accuracy. Neither scales 
could weigh the work more fairly. As far as the exam is concerned, all 
academics are born free and equal". And he goes on to say that: 


Now the method of examination lately adopted by the Boston 
School Committees, settles the question as definitively, what 
kind or quality of instruction has been given by the masters, as it 
does what amount or extent of proficiency has been made by the 
pupils (MANN, 1845, p. 332). 


The Boston School Committee implements the exams. "The tests 
used to be just to evaluate students, but now the test of students is used to 
evaluate teachers and, in fact, the entire educational system. 
(GUTTING, 1980). 


7 American educator and lawyer the first great American advocate of public education, who believed ina 
democratic society where education should be free and universal. 
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Only in the 20th century did Ralph Winfred Tyler® (1934) propose the theme 
"educational evaluation" as a verification that the proposed objectives were 


met. 


The Tyler principles described Kliebard (2011) in an article at the 
University of Wisconsin-Madison USA, describe that educational goals must 
be achieved in a curriculum. For this to happen, precepts are described and 
elaborated around four central questions that Tyler considers relevant to be 


answered; for the curriculum development process to develop: 


1. What educational goals should the school pursue? 


2. What educational experiences can be offered to 
achieve these goals? 


3. How can these educational experiences be organized 
efficiently? 


4. How can we determine if these objectives are being 
achieved? 


These questions can be formulated in the four-phase 
process through which the curriculum is developed: 


stating objectives, selecting -experiences, organizing - 
experiences and evaluating. 


Tyler’s principles are, in essence, an elaboration and 
explanation of these phases. The most critical phase in 
this doctrine is obviously the first, since all the other 


phases are based on the statement of objectives. In 


Tyler’s words: -lf we are to study an educational program 
systematically and intelligently we must first have 
absolute certainty as to the educational objectives to be 
achieved, (KLIEBARD, 2011, p. 24). 


According to Tyler (1950) the evaluated objective contemplated three 
sources: studies about the student, studies about the society and suggestions 
offered by the specialists in the content. The data obtained from these sources 


should pass through the psychological and philosophical sieves after exams. 


Thus, according to Kliebard (2011) educational evaluation was born, as we 
know in our schools today. 


8 American educator Ralph Winfred Tyler was a thinker of great importance for world education, especially 

with regard to curriculum assessment and development. Before Tyler, evaluation was seen only as a 
technical matter for measuring learning outcomes that always served as a comparison with other groups 

of students. The influence of psychometry and docimology, which focused on measurements and testing, 

was very strong. (ALVES & SARAIVA, 2013 p.1) 
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2.2 THE EDUCATIONAL ASSESSMENT AND ITS DEVELOPMENT 


Authors such as Daniel L. Stufflebeam and Anthony J. Shinkfield (1993) 
carried out studies on the capitalist production society, involving applied tests and 
examinations. Egon G. Guba & Yvonna S&S. Lincoln (1989) are authors who 
describe the generations or ages of evaluation. We still have Clarence J. Karier 
(1992) and S. Bowles & H. Gintis (1981) who made a link between economics, 
history and education and assessment. 


The researchers emphasized the importance of various assessment models 
in the educational system, particularly by mixing several qualitative and 
quantitative models of the same system, which show (slightly) different behaviors 
based on different assumptions. Typically, qualitative models are used to create 
the basis on which quantitative models can be understood and explained (e.g., 
Fredriksen and White [2002]; Sime [1998, 1996]; Sime and Leitch [1992 White 
and Frederiksen [1990]). The usefulness of this approach is generally proven, 
showing that students acquire a better understanding of the phenomena and that 
they have fewer misconceptions. In a study on the subject, Sime (2002) adopts 
a different perspective to evaluate the usefulness of this approach, asking 
students what they found useful for them while working on problems using various 
models. It was found that students easily accepted qualitative models and 
considered them valuable. 


In this sense, Barbier (1985) defends that, in the view of many authors and 
practitioners of evaluation, it is subjective. Thus, we have: "student evaluation, 
measurement, test, program evaluation, school staff evaluation, school 
accreditation and curriculum evaluation" (KELLAGHAN; STUFFLEBEAM; 
WINGATE, 2003, p. 1). 


As a reference in this research we clarify that educational evaluation is 
understood as several practices and procedures at different levels in the 
educational system, thus, described by Thomas Kellaghan, Daniel L. Stufflebeam 
and Lori A. Wintage (2003) in "International Handbook of Educational Assessment" 
free translation: "International Handbook of Educational Assessment". 
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With the intention of bringing a greater understanding of the subject, which 
is now being developed in this research, several authors and thoughts are cited 
to format a concept. We also find in the "International Handbook of Educational 
Assessment"? the description of the Theory and Meteorology of Assessment 
proclaimed by Scriven (2003) characterizing the theory of evaluation in two ways: 
The normative theories emphasizing what the evaluation should do or be, or how 
it should be conceived or defined. And the descriptive theories about which 
evaluations exist (classification theories) and what they actually do or have done, 
or why or how they did it (explanatory theories). So we have: 


[...| evaluation in formation provisionally designated as a 
deliberate and socially organized act ending in the production of 
a value judgment, if we are interested in the historical conditions 
of its appearance and development, we find that everything 
happens as if there were not one history of evaluation, but two, 
relatively parallel (CASTRO, 2020, p.68 apud BARBIER, 1985, 
p. 34, emphasis added by the author). 


An evaluation should, by definition, lead to a specific type of consideration 


- about merit, value or meaning - usually expressed in the language of good/bad, 
better/worse, good/bad, elegant/bad, etc. Describing such a situation Scriven 
(2003) explains. This restriction requires that evaluations - both in daily life and 
in scientific practice - involve three components: (i) the empirical study (i.e., 
determining raw facts about things and their effects and perhaps their causes); 
(ii) collecting the set of perceived and defensible values that are substantially 
relevant to the results of the empirical study, for example, through a needs 
assessment or a legal opinion; and (iii) integrating the two in a report with an 


evaluative claim as a final consideration. 


Most evaluators are driven by the changes forming a fad in the conception 
of the word, such changes in evaluations are determined by the amount of public 
money involved in the assessment. These changes are linked to results-oriented 


evaluations that have only produced an impact on society, are full of flaws, and 





° International Educational Assessment Manual - Educational assessment covers a wide range of 
activities including student assessment, measurement, testing, program evaluation, school staff 
evaluation, school accreditation and curriculum evaluation. It occurs at all levels of education systems, 
from individual student assessments by classroom teachers, to school and district assessments, to district- 
wide program assessments, to national assessments and comparisons of student performance levels. 
(Kellaghan & Stufflebeam, 2003). 


23 


are directed away from the model of good evaluation; they do not look for side 
effects that only demonstrate the results achieved or desired. 


Thus the evaluation is not only a process that determines facts about 
things, it must include its effects. By definition the assessment model must lead to 


a specific type of consideration, having values categorized as good or bad, better 


or worse, good or bad, etc. Thus described by Michael John Scriven’? (1985) 





Phase 1 - The Study 


Phase 2 - The Collection 


Phase 3 — The Conclusion 





(i) the empirical study (i.e., 
determining raw _ facts 


(ii) collect the set of perceived 
and defensible values that are 


(ili) integrate the two in a report 
with an evaluative claim as 


about things and_ their final consideration. 


effects and perhaps their 


substantially relevant to the 
results of the empirical study, for 











causes) example, through a_ needs 
assessment or a legal opinion; 
and 





Box 1 - Phases of Assessment (Prepared for this Study) 
Source: SCRIVEN (2013) 


In phase 1 of the study, according to the definition described in chart one, 
we find only the determination of the raw facts about things and their effects and 
indicating the causes. In phase 2 in the collection, the perceived and defensible 
values are considered relevant for the result, finally in phase 3, in the final 
consideration the report must integrate the two previous phases, presenting 
effects and causes as well as perceived and defensible values relevant for the 


result generating a lesson. 


The system presented in table 1 in this study, prepared by Scrivem (2013), 
describes a sequence that addresses the 3 phases of the evaluation. The 


qualification as value are mentioned in the phases being concluded in phase 3. 


2.2.1 Evolutions of Educational Assessment 


According to Castro (2020) some authors, among them Escudero Escorza 
(2003); Guba; Lincoln (1989); Stufflebeam; Shinkfield (1985); Jean-Marie Barbier 
(1985); Correia, [s/d] and Dias Sobrinho (2003); describe an evolution of 


educational evaluation. 


10 
Michael John Scriven states that "assessment is a type of inquiry, with basic logic and a general pattern of 
reasoning". It places the evaluation clearly in the field of knowledge production, using, to demonstrate this 
facet, the distinction between two types of complementary reasoning: the general logic and the applied 
logic of the evaluation ”. (PEREIRA & SILVA, 2015 p. 368) 
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(1800-1900) 


Authors Guba & Lincoln Stufflebeam & Cabrera & Barbier 
Shinkfield Salvador 
Term Generations 5 seasons 3 Great 2. History of Assessment 
seasons 
Early 19th century Reference 3. Great times 
pont Reference point 
Tyler 
Previous A limited extension of 
evaluation practices in 
social forms 
pre-capitalists 
Name Reform Season A generalization of these 


practices at the time when 
the great modern 


educational systems are 
organized 




















The first The period Mann (1945) 
generation pre-tylerian 
"Measurement/Me 
The second Tylerian Period Tyler Season 
generation (1930- 1945) 
The third Season of 
generation innocence (1946- 
"Trial" (post 2nd 1956) 
war 1946) 
Realism Season 
(1957-1972) 
The fourth Professionalisation 
generation period (from 1973 





to the present day) 





Develope 





A movement of 
rationalization in 
contemporary times - 
neoliberalism 





Box 2 - Evolution of the Assessment Activity 
Sources: Castro (2020, p. 97); Escudero Escorza (2003); Guba; Lincoln (1989); 
Stufflebeam; Shinkfield (1985); Jean-Marie Barbier (1985); Correia, [s/d] and Dias; Sobrinho(2003) 


Some authors agree with the evolution of evaluative activities, in generations, 


phases and times, evolving from Capitalism to Neoliberalism. 
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2.2.2 Evolution of the Assessment in the United States 


According to Viana (2003 p.63 ) "Tyler's nuclear thinking is found in the 
article - [General Statement on Assessment], which has become a classic in the 
genre and continues to be validated in the present, although it was published in 
1942. In the General Statement on Education "’, the foundations of Evaluation and 
Education in the United States were laid and influenced generations of educators 
Stake (1967); METFESSEL, N.S. and MICHAEL, W.B.(1967); Scriven (1967). 


We also found in the USA in 1958 "The National Defense Education" 
known as NDEA" - National Law in defense of Education. PL 85-864 Act of 
1958. In its description are the Objectives: "Loans to Students in Higher 
Education Institutions (Title Il); Financial Assistance to Strengthen Science, 
Mathematics and Modern Language Teaching (Title Ill); National Defense 
Education Scholarships (Title IV); and Guidance, Counseling and Testing: 
Identification and Encouragement of Able Students. (Our Griffon) Officially 
institutingthe Educational Test (Annex | - NDEA). 


According to Cronbach (1963 apud STUFFLEBEAN; SHINFIELD, 1993) 
when describing the "Curse improvement threugh Assessment’ in 1963, she points 
out the importance of evaluating the actions carried out in the educational 
environment, influencing works such as Scriven’s (1967) and Stake’s (1967), 
which, in turn, had a profound impact on the practice of educational evaluation. 


In the 1970s several texts on evaluation emerged, the most important of 
which were Educational Assessemnt and Policy Analysis (EEPA) [...] "educational 
assessment requires the description of results; thus, certain concerns with 
educational measures are for the production of accurate scores in order to 
compare individuals, or the comparison of mean scores of different courses [...]" 
(VIANA, 2003 p. 69)."Studies in Assessment"; "Assessement Review", all quoting 
Tyler (1942). 


11 Ralph W. Tyler developed the first systematic approach for educational assessemnt. This evolved from 
his work in the 1930s and early 1940s. Shinkfield (1985). 





12 NDEA represented an important step in expanding education opportunities for those interested in 
pursuing a career in science, mathematics or foreign languages. https://www.ida.org/research-and- 
publications/publications/all/t/th/the-national-defense-education- act-of-1958-selected-outcomes 
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3 THE ASSESSMENT STATUS 


In this Chapter we will describe how the State has become an evaluator of 
educational practices and educational programs. The evolution of State 


intervention in education and its governance in public education actions. 


3.1 REGULATION AND ITS POLYSEMY 
Aiming at a more detailed understanding of what the Assessment State is, 
Ball (2018) describes that "[...] is the process of (re)distributing responsibilities 
among educational actors and is understood as public action with a complex 


and multidimensional character". 


Trajetéria Go sistems de ensinoe 
das instituicdées 


Contexto sociale relacions! 





Figure 1. Analysis of the processes responsibilities among educational actors 
Source: BATISTA (2014, p. 2) 


Still in Batista (2014) we find some authors describing the "assessment 
state" (NEAVE, 1988; BROADFOOT, 2000) and "quasi-market" (WHITTY, 1996; 
MAROY, 2005) have been the concepts mobilized to summarize the emerging 


regulation models. 


The first refers to an a posteriori evaluation of results, in a context 
of remote control pilotage by the state, which has control over the 
definition of the objectives of the system and the gauging of its 
quality and leaves to the lower authorities the means to meet the 
goals previously set. This concept is associated with the notion 
of accountability, through control devices such as_ school 
inspections. (BATISTA, 2014, p.2). 


Thus we have the concept of REGULATION in Barroso (2005, p. 728) 


"being the operations that allow, in the use of material or in an operation with 


troops, the best result, the best use and combination of joint efforts, 
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before an objective, to achieve it in the best conditions and with the least 


effort". 


In this context, the educational domain by the State emerges coining 
the term "regulation" and associated in a broader and more objective way, 
although symbolically of intervention in public policies. Being a direct object 
of control, involving procedures linked to quality and effectiveness of results, 
to regulation and implemented as a practice of administrative innovation in 
bureaucratic control, finally evidencing the intervention of the State in 
education. 


We still have Dias Sobrinho (2002) apud Oliveira (2011, p. 54) that 
describes the "EVALUATING STATE‘? [...] in this phase the State 
abandons the role of provider of benefits and services, with a view to 
compensating citizens for existing social problems assumes that of controller 
and controller". Complementing this, Oliveira (2012) reinforces that Afonso 
(2005) also describes the model of the EVALUATING STATE as being: 


[...] the competitive ethos assumed by the neoliberal 
state. In the forms influenced by neoliberalism, the state 
assumes market logic, importing private management 
models into the public domain and emphasizing the 
products (results) of educational systems (OLIVEIRA, 
2012, p. 54). 


In the last century, a large number of conservative reforms have 
proliferated in the U.S. and England, among them the centralization of 
educational evaluation, this search for control, has made it possible to 
implement more effective instruments producing consistent changes in social 
sectors such as education. These "controls" were understood as the 
evaluating state exercising REGULATION. According to Barroso (2005), in 
conceptualizing regulation, he declares that they are controls associated with 
autonomous but interdependent elements. The Locus in which the 
intervention exercises legitimate authority (usually state). In GAMBLE (1994) 
we find the statement that the state did not fail to strengthen its coercive 


power, nor did it loosen its role of social regulation. 


13 Authors quoting: Whitty(1996); Neave(1988); Dias Sobrinho(2002); Broadfoot(2000); Afonso(2001); 
Maroy(2005); Oliveira(2012); Batista(2014); Yonnoulas, Souza & Assis(2009); Queiroz(2011); Ball(2018). 
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These various meanings seen as "regulation" by an Evaluation State 
form a mosaic of thoughts, where in an apparently paradoxical way the State 


promotes its own retraction while expanding its participation in education. 


3.2 THE CONFIGURATION OF THE EVALUATOR STATUS 


There is a consensus among authors Afonso (1999; 2000; 2001) 
Bernstein (1996); Oliveira (2012); Barroso (2005); Batista (2014); Neave 
(1988); Maroy (2005) that a normative framework based on the "obligation of 
means" for another, based on the "obligation of results". This evolutionary 
trend in educational policies in Canada, the United States, France and the 
United Kingdom has resulted in a regulatory state". (LESSARD, BRASSARD 
& LUSIGNAN, 2002, p. 35 ). 


Several works also cite the 1980’s as a source of public policy where 
there was a transformation in the purpose of the evaluation, ceasing to be a 
routine focused on the maintenance of the system, to become a strategic 


evaluation, with the objective of analyzing the present and regulatory situation. 


We found that the configuration of the evaluating State has evolved 


from a centralizing system to a decentralizing one, seeking greater control. 


The rise of the Evaluative State can no longer adequately 
be accommodated in terms of the Manichean dichotomy 
beloved of comparative educationists since the 1930s, of 
systems that are centralized versus systems that are 
decentralised. [...]. Thus, in systems based on 
decentralisation, the Evaluative State appears as a step 
towards greater central control and, in those based on a 
higher degree of centralism, it is perceived as giving rise 
to greater flexibility and hence greater decentralisation 
(NEAVE, 1988, p. 11).14 


The decentralization practiced in the "regulation" context is a step 
towards greater control, but it induces a perception of less centrality and 


greater flexibility. 





14 Decentralization, seeking greater control. The rise of the Evaluation State can no longer be adequately 
accommodated in terms of the Manichean dichotomy, loved by educators since the 1930s, of systems 
that are centralized versus systems that are decentralized. [...]. Thus, in systems based on 
decentralization, the Evaluation State appears as a step toward greater central control and, in those based 
on a greater degree of centralism, is perceived as generating greater flexibility and, therefore, greater 
decentralization (Neave, 1988 p. 11 ) 
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After carrying out a comparison between the educational systems in 
England and France with different administrative traditions, it was found that 
the evaluation was being used as the same differentiating mechanism in the 
two systems, as reported: 


While the two countries starting from radically different 
administrative traditions [...], it is increasingly possible to 
pick out similar trends in the two countries [...]. 
Educational activity in both England and France is 
closely controlled by prevailing assessment procedures 
(BROADFOOT, p. 218) 1°. 


In this observation identified by Broadfoot (2012) in the English and 
French system, both with differentiated management, brings a legal obligation 
to respond to higher authorities, through inspections and external evaluations 
to monitor results. It is called "Government-based accountability" '° This 
practice inserts the argument that the professional qualification of the teacher 
is no longer sufficient to supply the quality of the system. 


The Model described as an Evaluation State that uses accountability 
has an impact on the actors, by using the act of evaluation as a moral 
dimension, resulting in the acceptance of a culture, which | insisted on the 
incorporation of dominant values and the need to present results and be 
accountable. 


The evaluation is therefore seen as a tool of the system that is 
associated with performance, receiving symbolic or material incentives in view 
of the results achieved. Consolidating that evaluation is a state public policy, 
with regulatory purposes. 


15 Although the two countries start from radically different administrative traditions [...], it is increasingly 
possible to detect similar trends in the two countries [...]. The educational activity in England and France 
is closely controlled by the current evaluation procedures (BROADFOOT, p. 218) 


16 In the field of education, accountability tends to be synonymous with new forms of outcomes-based 
regulation (MAROY, 2006, 2008). 
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3.2.1 The Political Origin of the Evaluating State 


In 1940 a group of Austrian economists disagreed with the pressure exerted by 
the American and English governments on the rest of the world, identified as 
the "New Collective Policy"’’, among them Karl Popper, Ludwig Von Mises and 
Friedrich Hayek. From this infusion of ideas between a philosopher and two 
economists, one being a liberal and the other a left-wing economist, the Mont 
Pelerin'® Society was founded by Hayek to develop his theses, and thus was born 
the neoliberalism that had as its main idea according to Stedman (1974) to fight 
against state control with the following idea "central planning was impossible, 


because nobody, regardless of their intelligence, knew what people wanted. ” 


Considered the second wave of criticism against the state, it was 
the Chicago economist Milton Friedman who, in his 1951 article entitled 
"Neoliberalism and its Prospects," among other things, defended an 
"intermediate point" between the collective enemy and the excesses of 19th 
Century liberalism, coining the expression "market competition" that, according to 


him, would protect men from each other. 


Thus, the milestone of the birth of the Evaluation State (Regulation) 
is linked to the ideas of Neoliberalism, according to Yannoulas; Souza; Assis; 
(2009, p. 58) 


Evaluation programs are perhaps the most important and 
efficient mechanism for controlling and regulating the 
process of restructuring the role of the state in 
educational matters. While it is true that the evaluation of 
policies in education is not a novelty, the centrality given 
to the evaluation of public policies in contemporary times 
responds to the characteristics of the prevailing 
neoliberal logic, with the partial withdrawal of state 
responsibilities in social issues, and the encouragement 
of competitiveness in the market. In this context, 
knowledge is seen as one of the greatest assets of the 
State and also as a factor co-responsible for economic 
development. 


Considering public policies specifically in education as the central framework 
of regulatory control of the state, and encouraging market competitiveness, the 
logic of neoliberalism is based on practice. 





17 A movement created by the USA and England based on the Capitalism of John Maynard Keynes who 
postulated his theory in which he totally broke with liberalist principles. 
18 Miovement that creates Neoliberalism (https://www.montpelerin.org/ ). The Mont Pélerin Society isan 
international organization that emerged then in response to the social, political, intellectual and moral 
ruin that dominated Europe during and after World War Il. 
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The evolution of the political idea of state regulation is effective as a 


logic reigning in times of investment in countries that are undergoing changes, 


being convinced to this practice. 


In the late 1980s the interest in the issue of social policy evaluation 
demonstrated by the neo-conservative and neo-liberal governments 
of some countries began to be translated by the expression 
"Evaluation State". Guy Neave, professor of comparative studies 
on higher education policies at Twente University (Netherlands) 
consolidates the term. [...] the focus of the Evaluating State is not 
on pedagogical processes, but on the results of pedagogical 
processes (YANNOULAS; SOUZA; ASSIS, 2009, p.59). 


Understanding that few areas of activity escape the control exercised by 


the evaluating State (Health, Education, etc.). We also verified that this discussion 


happens especially in the area of Education. From the twentieth century on, 


evaluation became obsessive and unidirectional on the part of neoliberal 


governments, following party-political guidelines, carrying out a legitimization of 


democratic control called regulation. Thus, there would be a history of the 


practices of evaluation of people in formation, described below. 





Phase 1 - XV Century 


Phase 2 - Century 


Phase 3 - 20th Century 





i. "a limited extension of 


valuation practices in pre- 
capitalist social forms" 15th 
to 18th Centuries; 








ii. "a generalization of these 


practices at the moment when 
the great modern educational 
systems are organized" 18th 


and 19th centuries; 





iii. a movement of 


"rationalization" in 


contemporary times" 








Box 3 - History of practices of assessment of people in training 


Source: Barbier (1985, p.34) 


Although this approach of Barbier (1985) does not specifically coincide 


with the "facts" and fundamental aspects of the "periods" of Stufflebeam and 


Shinkfield (1993), the description of the history of the evaluation practices of people 


in formation and replaced by traditional models of professional and bureaucratic 


control, starting to be practiced by Capitalist Governments and reaching its peak 


in neoliberalism. 
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3.2.2 The quasi-market logic in education 


Some authors such as Appadurai (1996), Van Zanten & Ball (2000), Barroso 
(2006) and Antunes & Sa (2010), understand that the changes made by the 
regulation of educational systems in the last century are standardized in the concept 


of "quasi-market", comparin "Soublic and private schools and implementing aspects 
of "competitive logic"; they also found that in these changes other modes of 
regulation, although voluntary and horizontal, emerge among them the 
"collaborative logic". 


Still according to Barroso (2006) the concept of regulation is 
connected to the constructivist vision, structured in multiple and sometimes 
contradictory processes, presenting cognitive models and legislation in 
institutional regulation or norms, with local scope, resulting in individual or 


collective actions in face of a systemic standardization. 


Thus, the "evaluating state" Neave, 1988; Broadfoot, 2000 and 
"quasi-market" Whitty, 1996; Maroy, 2005 are concepts presented to summarize 
the emerging regulation models in education systems, being assigned 
competencies among the educational actors, based on evaluation mechanisms 
with the purpose of accountability and presenting results that will have as 


objectives the choice by families and exercise regulation. 


There are several views on the evolution of the concept of evaluation 
to "near-market" among these authors, some with a comparative approach, 
highlighting the main features of the model and the legislation produced in the 
country, others, however, present an analysis understood at the level of the local 
context, making a framework of the overall guidelines of educational policies. 
Finally we have authors who highlight the competitive logic and the multiple 
processes of local regulation at play. These reflections are based on the literature 
cited in this work and are made explicit in the framework where we highlight the 
evolution of these concepts of "Quasi-market" over the years. 





1° Financing education in developing countries: an exploration of policy options (WORLD BANK, 1986). 
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Author Tyler (1942) Cronbach (1963) 
B [...] evaluation methods are generally fixed | The evaluation aimed at the improvement 
A in written exams, when in fact they are | of curricula should not be confused, as 
S , hea ncks instruments of measurement and 
| valid, for classification purposes. obtaining scores. Evaluation is not 
C indispensable for it to achieve its 
objectives. 
Neave(1988) Stain(1988) Education Reform Act 
C A posteriori evaluation of results, in a | [...] that has changed the education system in 
O ; England, particularly by admitting parents’ 
N context of remote control pilotage by the | choice of school and increasing the autonomy 
C state, which has control over the definition | of schools, diminishing the traditional power of 
E of the objectives of the system and the | local education authorities. 
P measurement of its quality and leaves the 
T authorities inferior to the means to meet 
the goals previously set. 
Author Le Grand ( 1991) Levacic (1995) 
Cc [...] by a state financing services provided by a | [...] unequal capacity of families to make 
O variety of suppliers (public and private), who | informed choices about the quality of education 
N operate in competition for clients who, on the | provided and the not only private but also public 
C basis of information concerning the quality of | nature of the educational good. Particularly 
E services, exercise their right of choice. with regard to the entry of new suppliers 
P (schools), investment and quality of service. 
T 
Author Broadfoot (2000) Delvaux and Maroy (2009) 
Cc This concept is associated with the notion | The constitutional principle of freedom of 
O of accountability, through control devices | choice has resulted, on the one hand, in a 
N ; ; multiple and decentralized school initiative. On 
C such as school inspections. the other hand, it allows families to freely 
E choose their schools. The competitive logic has 
P traditionally been characterized by a "principle 
T of discretion”. 
Author Van Haecht (2004) Barzano (2009) 
C Logic guided by the educational offers and the | The encouragement of freedom of choice and 
O characteristics of the schools, at the same time | competition between schools was_ also 
N that they develop competitive strategies to | achieved by financing schools according to the 
€ attract families and students. number of students, as well as by greater 
E dissemination of information on the functioning 
P of schools, in particular through the famous 
T league-tables, which sort them according to 
the classification in national exams. (Our 
Griffon) 
Author Afonso (2010) Mangez (2011) 
Cc ia applies in cases where a set of [---] thus ensuring the coexistence and 
O mechanisms and devices. favour the | autonomy of the different suppliers. Cultural 
N and social transformations have accentuated 
C development of logic and values of | the mercantilist logic, to the extent that parental 
E comparison, competition and freedom of choles today epee! pia quiged BY ielglous 
P choice in the public system, as long as they do or political convictions and increasingly guided 
T by the educational offer and the characteristics 





not lead to privatization or are not introduced 





of schools 





Box 4 - Evolution of Near-Market concepts 
Source: Prepared for this work 





4 HISTORY OF ASSESSMENT IN BRAZIL 
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In this Chapter we will present the many evaluations that exist in 


Brazil, their evolution, the descriptive picture by time and the separation in the 


levels of education. 














STEPS ASSESSMENT - PERFORMANCE EVALUATION OF 
HIGHER EDUCATION ASSESSMENT LEARNING 

Stage 1 Investigate and IDEB - Basic 
understand the historical | ENEM - National High | Education 
background of School Exam; Development Index; 
Evaluations: 
PARU/CNRES/GERES 
in the IES; 

Stage 2 Description of PAIUB, ENCCE} A- National SAEB - Basic 
Programa de Avaliagao Exam for Certification Education Evaluation 
Institucional das of Youth and Adult System. 
Universidades Competencies; National Evaluation 
Brasileiras (1993); of Basic Education - 

Aneb 

to investigate, describe National Evaluation of 
and analyze the School Income - 
knowledge produced in Anresc. 
the ENC(1995) ANA - National 
National Course Exam Literacy Assessment 
and the ENADE (2004); 

Stage 3 Examine and analyze PISA - International 








the SINAES Evaluation 
processes; 





Student Assessment 
Program; 





Brazil Tasting (5th to 
9th) 

Provinha Brasil 
(Literacy of enrolled 
children). 








Box 5 - Stages of Assessment Brazil 
Source: Table developed for this work 


4.1 FUNDAMENTAL ASPECTS OF EDUCATIONAL EVALUATION 


According to Santos Filho (2016) in his research on antecedents of SINAES - 


National System of Evaluation of Higher Education, a fact called "Analysis of Evaluation 


Policies 


20 


that contemplates authors among them appears in 1960: Almeida Junior 


(2005a, 2005b); Zandavalli (2009); Rothen and Barreyro (2011); Santos Filho (2016), 
Dias Sobrinho (1996, 2003, 2010); Lima (2005); Rothen and Schulz (2006, 2007). It 


is the historical landmark of the Brazilian evaluation. According to Zandavalli (2009), 


the Military Coup that took place in 1964 inserted in Brazil a Capitalist economic 


policy and an Avaliative policy in the national educational system supported by 
the United States. 


20 AVA, PARU, CNRES, GERES, PAIUB and ENC evolution of Evaluation Policies in HEls. 
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In higher education the document "Assessment of Brazilian Higher 
Education the University Reform of 1968" outlines a profile of the level of Brazilian 
education, focused on vacancies. For researchers Zandavalli (2009), Zainko 
(2008) Santos Filho (2016) among others, the PAIUB?' - Brazilian 
Universities Institutional Evaluation Program of 1993 formatted and gave rise to 
SINAES - National System of Higher Education Evaluation as we have today. 

The Evaluation Program of Brazilian Universities in 1993 brings 
together authors and professors well known in the field of educational evaluation 
in Brazil. Zandavalli (2009) argues that the proposal of PAIUB was prepared by 
representatives of ANDIFES and presented to the MEC in 1993; being thought 
since the 1980s the Institutional Evaluation is defined in the 1990s having 
practical effects on HEls after approval of PAIUB by Law No. 2026/96. 

According to Dias Sobrinho (1996), the evaluation in Higher 
Education stands out for ideological parameters defined as a struggle in 
education, having basically two aspects. The first with bureaucratic technical 
characteristics and the other more participative/emancipatory, the evaluation 
practice seen in this way, as producing great impacts and transformation in the 
environment of HEI. On the other hand, the technical-bureaucratic influences 
directly on privileges and promotions and promotes the decentralization of social 
value inserted in the evaluation. 

Also according to Dias Sobrinho (1996 p. 20 apud CASTRO 2020 
p.141): "The university cannot let the ownership of the evaluation slip". As 
promoters of evaluation and self-evaluation, HEls are protagonists of the 
evaluation processes, inserting criteria focused exclusively on numerical 
(quantitative) results, to the detriment of social promotion and the valorization of 
research and extension. Thus, there is a conflict of interest when evaluating, and 
if there are good results, publicity and promotion of a ranking will happen, 
otherwise (unsatisfactory results), conflicts occur, with loss of funding and 


prestige with the academic community. 





21 In the PAIUB document, legitimacy is double: a) political, guaranteed with the participation and 
involvement of all, and b) technical, manifested in the methodological competence of its realization and 
the reliability of statistical data. The following principles guided the elaboration of the PAIUB: 1. Globality. 
2. Comparability. 3) Respect for institutional identity. 4. non-punishment or reward. 5. voluntary 
adherence. 6) Legitimacy. Continuity (BRAZIL, 1993). 
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4.2 ENC - NATIONAL COURSE EXAM 


The 1968 University reform implemented by Law 5,540/68 created a 
model of Higher Education that was contested by many and suffered severe 
criticism from educators. The UNIVERSITARY REFORM EVALUATION 
PROGRAMME - PARU (1983) was then created; which years later would 
result in the creation of GERES (1986) - Executive Group for the reformulation of 
Higher Education, which proposes, among other things, a new evaluation of 
higher education. The PAIUB - 1993 and two years later the National Examination 
of Courses (ENC - Law n. 9.131/95) was created, making HEls compulsory. The 
ENC is linked to the issuing of the diploma of the course the realization of the 


exam. 


The National Examination of Courses 22, also known as "Provao" was 
the fruit of the political movement called classic liberalism that foresaw among 
other things: The free market; Free competition and non-intervention by the State, 
founded in the late 1980s; the evaluation process was characterized as a 
mechanism that presented the growth of capitalism in the world. This evaluative 
model was associated with external evaluation and institutional self-evaluation, 
according to the international evaluation movements of the time. 

Formulated to manage the quality of higher education, the ENC 
establishes a cycle of annual evaluation to the graduates, minimizing the old 
model of control carried out by state intervention in the HEI, to verify the quality 
and social responsibility. The National Examination of Courses strengthens the 
market valuation as a mechanism to assess the value and continuity of 
operations, creating the massification and diversification of university education. 

Complementing the ENC, Decree 2.026/96 was edited, which 


defined parameters, metrics and analytical indicators for the evaluation of higher 
education based on the National System of Higher Education, which considered 


the Federation Unit, the type of HEls and their location; 





22 ENC has "principles": indicators of results, comparison of the performances of the institutions, ranking 
of the courses in the form of concepts. The performance of the undergraduate courses is inferred from 
the performance of their students (Gomes,2011) https://doi.org/10.1590/S0100-15742003000300007 
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4.3 ENADE - NATIONAL STUDENT PERFORMANCE EXAM 


With the objective of assessment the performance of scholars who 
have completed the higher education course and verifying the fixation of the 
program content foreseen in the curricular guidelines of the course, ENADE 
measures the competences and abilities of the trainees, in view of the 


professional formation and the reality of the market. 


INEP - Instituto Nacional de Estudo e pesquisas Educacionais Anisio 
Teixeira, is the manager of ENADE, integrating SINAES - Sistema Nacional de 
Avaliagao da Educagao Superior (National System for Higher Education 
Evaluation), also composed by Graduation Courses Evaluation and Institutional 
Evaluation. These evaluation systems are described in the following image: 










Ministério da Educacao Util 
itura i. liza como 3 
¢ Cultura (MEC) poses de Crotepclament orientador—, jects Greet 
Autarquia Federal vinculada ao — o> "ot me “ —» Recredenciamento ore (IGC) 
melhorar a 2 Tnstitucional ~—supervisio Comissées de 
qualidade do de IES avaliagao 
ES brasileiro INEP — Institucional Roelee Calculo pela 
t Prncexsoacie Autorizacao média do 
CONAES objetivo Responsavel pela pernlaase > Realizadas 
=. ’ operacionalizagio guage _ ——» Reconhecimento —> por Recursos \ 
supervisiio ; 5 
Coordena . erie ie _ oo Pedigorene Conceto 
supervisiona o SINAES — Pilares da < i? — Renovacio de Preliminar 
. avaliacio Avaliacio reconhecimento ae Considera as de Curso 
Propés a de Cursos Avaliagao __,* gM Corpo docente (CPC) 
- do buh in:loco aks ° 
estabelece a : 
Comissio Especial — utiliza —~+ Indicadores a ae 
de Avaliagio de Qualidade. "ecco 
do ES (CEA) 


Comissio Propria determinantes Esperado (IDD) 





de Avaliagao (CPA) . 
} Avaliacgio considera 
noes ‘ do Desempenho resultados da 
EE eee imcnnns | oeExtudantes ‘—* provafeita Hun 
ENADE pce 


Figure 2 - Mapping of the Brazilian higher education evaluation model. 
Source: Pereira, Machado and Araujo (2015, p.10). Authors’ elaboration. 


Note: The mapping illustrates the functions of the main responsible bodies: MEC, INEP, SINAES, 
CONAES, highlighted in bold. It allows identifying the relations between the evaluation elements: 
CPA, ENADE, IGC, CPC and IDD. Source: Pereira, Machado and Aratijo (2015). 

Figure 2 shows the three elements that make up SINAES, the 
metrics and parameters involved in measuring quality in HEIs, under the 
supervision of CONAES - the National Commission for the Evaluation of Higher 
Education, which operates the evaluation system establishing guidelines for the 


evaluation process. 
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4.3.1 Descriptive Analysis of ENADE 


According to Pedrosa; Amaral; Knobel (2012) and Verhine; Dantas 
and Soares (2006); the Brazilian evaluation model was designed by law and built 
over time, becoming a complex system and difficult even for its operators to 
understand. 


Such "Avaliative Pillars", called in figure 2 by the authors Pereira, 
Machado and Aratjo (2015) are: Evaluation of Courses; National Student 
Performance Exam - ENADE and Institutional Evaluation, were designed to cover 
Teaching, Research and Extension. 


According to Hussein (2009); Webb et al., (1966), they state that 
there is a trend that justifies multiple approaches to investigative methods in the 


social sciences, and can combine quantitative and qualitative. 


Therefore, we understand that ENADE gave continuity to the 
National Examination of Courses by preserving the metrics approach of teaching, 
comparing the HEls with each other and disseminating the result, thus inserting 
the logic of market education. The academic community at first linked ENADE to 
the quality of teaching and the construction of a performance index; over time, 
after a deeper analysis, they concluded that it was one more regulation in 
education, the same thing that happened with ENC. 


According to Rothen; Barreyro (2011),ENADE emphasizes the 
ranking of the IES promoting a market competition and establishing a competition 
bringing a similarity with the National Course Exam. The INEP, in describing its 
importance, emphasizes the development of skills and competencies in: and 
understanding, synthesis, content integration; resembling the ENC, and starting 
by sampling from 2005 to 2008 among entrants and graduates. It is a mandatory 


curricular component from 2009. 


4.4 SINAES - National System of Higher Education Assessment 
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Law 10.861 of 2004 of the MEC instituted SINAES with the main 
objective of a diagnostic, formative and regulatory evaluation of Higher Education 


Institutions. Described in figure 2 having three basic components: (1) Institutional 


Evaluation; (2) National Student Performance Exam; and (3) Course Evaluation; 


the management of the IES, the Faculty, the Facilities, etc... are also evaluated. 


The proposal of SINAES is to evaluate everything. 


Modalidades 
de avaliacgao 
Avaliagao dos cursos 


Avaliagao dos alunos 


Avaliagao dos 
professores 


Avaliagao didatico- 
pedagdgica do ensino 
Avaliagao de servido- 
res técnico-adminis- 
trativos 

Avaliagao das carreiras 





Aspectos a serem avaliados/procedimentos avaliativos 


Avaliagao por parte de especialistas segundo areas do conheci- 

mento. 

Avaliagao dos recursos fisicos, financeiros e pedagdgicos das 

instituigdes de ensino superior (IES). 

Indicadores de eficiéncia de diversos tipos. 

Avaliagao do curso pela demanda dos alunos e por sua qualificagao 

anterior. 

Avaliagao comparativa de formandos, por meio de testes padro- 

nizados. 

Avaliagao de oportunidades de trabalho, que buscam analisar a 

importancia do desempenho educacional na determinagao das 

carreiras. 

Avaliacgao dos indicadores de pesquisa e dos cursos de pos- 

graduacao. 

Avaliagao reputacional dos professores, por parte de seus pares e 
parte dos alunos. 

Correlagao entre o dominio do conteudo das mateérias ministradas 

e as habilidades didatico- Sgicas do professor. 

Verificagao do desempenho funcional, tendo em vista a subordi- 

nagao das atividades administrativas as atividades académicas. 


Comparagao entre carreiras, no que toca a reputagao, demanda, 
custos e resultados proporcionados aos formandos. 





Box 6: Modalities of Evaluation Established by the Ministry of Education 
Source: ZANDAVALLI (2009, p. 409). 


The initial proposition of the MEC evaluation system dealt with six 


different areas each in different modalities: (1) Course Evaluation; (2) Student 


Evaluation; (3) Teacher Evaluation; (4) Didactic-Pedagogical Evaluation of 


Teaching; 


(5) 


Evaluation. 


Technical-Administrative Servants Evaluation 


(6) 


Career 


The evolution of evaluation systems in Brazil has formed a mosaic 


of information, complex and difficult to understand by its users (IES Managers, 
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Teachers and Students), presented in figure 2. The legislation of this evaluation 
structure is described in chart 7 below: 





HISTORY OF THE ASSESSMENT OF HIGHER EDUCATION IN BRAZIL 











YEAR PROGRAM HARACTERISTI 
1983 University Reform First proposal of Higher Education Evaluation; 
Evaluation Program 
(PARU); 
1985 Executive Group for Higher | It presented the evaluation as a counterpoint to the 


Education Reform (GERES); | autonomy of HEls, giving relevance to the individual 
dimensions, whether of students, courses and institutions, 
although the concern with the institutional dimensions 














remains; 

1993 Institutional Evaluation Conceived the self-evaluation as an initial stage of a process 
Program of Brazilian that, once triggered, extended to the entire institution and 
Universities (PAIUB); was completed with the external evaluation; 

1995 National Examination of Carried out by graduates, emphasizing results, productivity 
Courses (ENC) - Law and efficiency, with performance control against an 
9.131/1995; established standard and with accountability; 

2001 National Education Plan It establishes that the Union will instruct the National 
(PNE) Law 10.176/2001; Evaluation System; 

2004 National System of Its purpose is to improve the quality of higher education, to 
Evaluation in Higher guide the expansion of its offer, to permanently increase its 
Education (SINAES) Law institutional effectiveness and academic and __ social 
10.861/2004; effectiveness and, especially, to promote the deepening of 


the social commitments and responsibilities of institutions 
and higher education; 

Box 7 - History of Higher Education Assessment in Brazil. 

Source: Souza (2013, p. 23) 

















The authors Morgan and Shahjahan, (2014); Richardson and 
Coates, (2014); Shahjahan, (2013); Shahjahan and Torres (2013) describe an 
assessment called Assessment of Higher Education Learning Outcomes 
(AHELO)?3 internationally known for evaluating the performance and effectiveness 
of recent graduates, applied by the OECD - Organization for Economic 
Cooperation and Development, performing an accreditation of the quality of 
higher education, not yet contemplated in the Brazilian Assessment System. 


According to Barzelay (2014); Hanushek, Link and Woessmann 
(2013); Lenkeit and Caro (2014); Morgan and Shahjahan (2014); Soh (2014) point 
out that an evaluation system is in place to measure effectiveness, believe 


what has been taught and legitimize what has been proposed in the content. 





23 AHELO - Evaluation of higher education learning outcomes will allow comparison between higher 
education institutions in all countries. This is done in the context of a massive growth of higher 
education and increased international attention to comparisons between institutions. 
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Thus, we have the evaluation as a mechanism that affirms quality first of all; it 
realizes an educational sense among students and finally presents a measure, 
establishing a ranking among the evaluated institutions encouraging competition 


among the Institutions of Higher Education. 
4.5 THE MANY ASSESSMENT OF THE SINAES 


The National System of Evaluation of Higher Education described in 
figure 2 - Conceptual Mapping of the Brazilian Higher Education Evaluation 
Model, presents itself as a complex system where quality indicators are 
measured by internal and external determinants. In the external evaluation 
according to INEP (2010); Ristoff and Giolo (2006) there are instruments that 
specifically guide this practice in self-evaluation reports, with the requirement of 
a minimum standard for quality. 

The Ordinance 1.027 of 2006 instituted the CTAA - Technical 
Commission of Evaluation Follow-up, being a collegiate organ, which by 
definition, deliberates evaluations "in loco" referenda evaluators and receives 
resources from evaluation processes of DAES - Board of Evaluation of Higher 
Education, BASis - Bank of Evaluators of Sinaes and System of Evaluation of 


Schools of the Government. 





Aloe Instituigdes 
a : 
Padres de Qualidade <—— obedece Externa. ——___+ a > CTAA —— syatiandd 
k F 
| aed Cursos 
des selecionando 
| ¢ nomeando 
signee composto 5 cixos 
¢ avaliagao incipai Comissdes formadas 
por principais 
de Avaliagio por ~, Beohes 
| externos 
Planejamento ¢ Com (51) Politicas | | 
Ayaliagto TES |= ————._ indicadores ———» académicas compostas . 3a8 sclecionados de 
(5 indicadores) i Tig (13 indicadores) por integrantes = 
Politicas de = Infraestrutura ISA ies — straves dé BASis 
gestio institucional fisica 4——— indies I (Banco de 
(8 indicadores) (9 indicadores) (16 indicadores) ISA curso avaliadores) 


Figure 3 - External evaluation in the Brazilian higher education evaluation model. 
Source: Adapted from Pereira, Araujo and Machado-Taylor (2018, p. 8). Authors’ elaboration. 


Figure 3 shows 5 main axes with 51 quality indicators inserted in the 
evaluation instruments. The NEP Ordinance 208 of 2016 establishes criteria for 
selecting external evaluators. Two criteria are contemplated: ISA-ies Assessors 
Selection Index for Higher Education Institution and ISA-courses Assessors 


Selection Index for Higher Education Courses. 


Variaveis 
(Experiéncia em:) 


Peso 
(% da 
nota) 
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Indicadores de 
Experiéncia 
(medidas em anos 
completos) 


Utilizada 
no indice 





ExpAC 


(Avaliacao de Cursos) 
ExpNDE 

(Nucleo Docente Estruturante) 
ExpDOC 

(Docéncia no Ensino Superior) 
ExpGC 

(Gestao de Curso) 


Pub 
(Produ¢ao Cientifica) 


cc 

(Conceito de Curso) 
ExpAl 

(Avaliacao Institucional) 
ExpNDE 


(Nucleo Docente Estruturante) 


ExpCPA 


(Comissao Propria de Avaliacao) 


ExpGA 


(Gestao Académica) 


Pub 
(Produ¢ao Cientifiea) 


cl 


(Conceito Institucional) 





4 


10% 





10% 





Conducao ou participacio 
em avaliagoes de curso 


Presidente ou membro integrante 


Em cursos de graduacgao 
ou pos-graduacio 
Coordenag¢ao ou assessoria de cursos; 
chefes de instituto 
Publicagées por ano 
nos ultimos trés anos 
Conducio ou participacao 
em avaliacées de curso 
Condugiio ou participagao directa 
em avaliagdes institucionais 


Presidente ou membro integrante 


Coordenagao, presidéncia 
ou membro integrante 
Cargos de reitoria, pro-reitoria, 
diretoria, gestao académica 
ou gestao de qualidade 


publicagées por ano nos 


ultimos trés anos 


Emitido por comissio ad Soc do 
INEP para a LES como resultado 
da ayaliacgao de Credenciamento/ 

Reeredenciamento 


Box 8 - Variables that make up the Evaluator Selection Indexes: ISACURSO and ISAIES. 
Source: Based on Portaria n. 208 INEP (Brazil, 2016c). 


Described in Ordinance 208 INEP 2016 the statistical modeling for 
the calculation of evaluators has individual weight of each index** (ExpAl 
(10%) + ExpNDE (20%) + ExpCPA (20%) + ExpGA (80%) = 80%). For calculation 
purposes they are using the experience in NDE - Nucleo Docente Estruturante 


and CPA - Comissao Prépria de Avaliagao, both are internal commissions of IES. 


In this context it establishes a ranking among the candidates for external 


evaluation commissions, following the equations: 


Equation]: ISA"curso"=ExpAC(10%)+ExpNDE(25%)+ExpDOC(20%)+ExpGC(25%)+Pub(10%)+CC(10%) 
Equation 2: IAS"ies: = ExpAl(10%)+ExpNDE(2%)+ExpCPA(20%)+ExpGA(30%)+Pub (10%)+Cl(10%) 


*4 The insertion and modification of indicators (indexes) are carried out according to multiple factors, 
qualitative and quantitative. TECHNICAL NOTE NO. 16/2017/CGACGIES/DAES. 
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4.5.1 Performance Indicators based on student results 

Ordinance number 8 of 28 April 2016 issued by MEC lists new 
quality indicators for Higher Education, generating quality in Higher Education; it 
also created the GTAES - Working Group for the Evaluation of Higher Education 
Performance, with the purpose of implementing and creating a methodology that 


would support new performance indicators. 


Baseado nos resultados do ENADE: considcrando o nivel de 
proficiéncia dos concluintes. Deverd expressar o valor absoluto da 
média do desempenho estudantil por curso. 


Sera calculado com base nos resultados do ENEM ¢ do ENADE. 


Acompanhamento da trajetéria dos ingressantes. A proposta é 
utilizar as taxas de permanéncia, desisténcia ¢ conclusio. 


Calculado com base no censo sobre o regime de trabalho, titulaciio ¢ 
permanéncia docente no curso. 


Substituto do CPC, utilizando os insumos IDE, IDD, ITE e IDCD. 


Pura cada IES, calculado pela média ponderada das matriculas, do 
conjunto de LDC dos respectivos cursos de graduagiio. 


Sera calculado com informacdes do censo ¢ dos relatérios de 
avaliagdo externa in /oco do SINAES. 


Vai substituir o TGC, com o objetivo de analisar o desenvolvimento 





institucional em relugao uo ensino, pesquisa e extensao. 
Box 9 - New Performance Indicators. 
Source: Portaria n.8 MEC Brasil (2016d, p. 11-12). 
The Performance Indicators Created by GTAES; IDD: Indicator of the 
Difference between the Observed and Expected Performances; ITE: 
Undergraduate Students’ Trajectory Indicator; IDCD: Faculty Development 
Indicator; IDC: Undergraduate Courses Performance Index; IIDC: Indice 
Institucional de Desempenho dos Cursos; IDEx: Indicador de Desempenho de 
Extensao; IDI: indice de Desempenho Institucional; ENADE: Exame Nacional de 
Desempenho dos Estudantes; ENEM: Exame Nacional do Ensino Médio; IES: 
Instituigoes de Ensino Superior; SINAES: Sistema Nacional de Avaliagao da 
Educacao Superior; IGC: indice Geral de Cursos. The indicators have the 
function of representing the results of students of higher education in the country. 
Pereira, Araujo and Machado-Taylor (2018) describe these quality 
performance indicators, elaborated by GTAES created by MEC 2016 Ordinance 
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n.8, in the form of a diagram figure 4, Indicates 8 (eight) new indicators/indexes, 
previously described, gauging the’ results of students involving 
ENEM/ENADE/SINAES. 








ea ee f Indi nite rb ——> age 
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Indice de 


Figure 4 - New quality indicators of Brazilian higher education. 
Source: Adapted from Pereira, Araujo and Machado-Taylor (2018). 





These new indicators/indexes can and should generate a better 
result in the SINAES evaluation model. They have been reformulated and 
indicated for the minimum qualification of evaluators; elements that make up the 
external evaluation process. In general, the set of actions that involve the 


evaluation practice in higher education were remodeled. 


Ordinance 1.027 of 2006 describes 51 (fifty-one) types of external 
evaluation distributed over the 5 axes, as shown in figure 3 prepared by CTAA. 
These quality indicators remained in force in Brazilian education until the edition 
of MEC Ordinance n.8. 


According to Pereira, Araujo and Machado-Taylor (2018) the 
evaluation structure in Brazil was transformed after the reform and implemented 
by Ordinance n.8 MEC of 2016 and Ordinance 208 INEP of 2016. 80 (eighty) 
indicators/indices adopted in the evaluation dimension of the courses were 
outlined: Organization didactic-pedagogical, Teaching Staff, Tutorial and 


Infrastructure. 
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5 ASSESSMENT IN THE EUROPEAN COMMUNITY 


In this chapter we will describe how the countries of the European 
Union deal with Education and its educational assessment. Its Metrics and 
Modeling in an Integrated Educational System; we will highlight among the 
Member States Portugal and how each one maintains an isolation between itself, 
applying different rules, qualifications and curricula. Each Member State intends 


to preserve its diversity. 


5.1 EDUCATION IN THE EUROPEAN UNION - EU 

The Maastricht Treaty gave birth to the European Union * EU in 1993, and 
was revised in the Lisbon Treaty that came into force in 2009. With its 
headquarters in Brussels as coordinator; the 28 member countries of the EU, the 
EACEA - The Education, Audiovisual and Culture Executive Agency is 
responsible for the management of the Education, Culture, Audiovisual, Sports, 
Citizenship and Volunteer programmes. It also produces studies, offers online 
resources, coordinates and manages the Eurydice Network?°, an institutional 
network set up in 1980 on the initiative of the European Commission that collects, 
updates and disseminates reliable and comparable information on policies and 
education systems across Europe, providing information and analysis on 
European education systems and their policies. 

In 2014 the European Commission carried out a stocktaking exercise 
among the Member States reviewing the progress made from 2012 to 2014 and 
defining new priorities until 2020. According to Neves (2018) this is a very critical 
view of the "way" of doing politics and the apparent "freedom" of the European 
Union to define its own strategies within the Education systems. 

Novoa (1998, 2005); Ozga, J. & lingard B. (2007) define the practice 
of "form" and "freedom" employed by the EU as different models of political 





25 With the entry into force of the Treaty on European Union, the EEC becomes European Community (EC). 
26 Eurydice provides a wide source of comparable information on Europe's education systems and 
educational policies, as well as a wide range of comparative analyses on various topics relating to 
education systems. The network supports European cooperation based on documented data in education 


and lifelong learning. www.europa.eu 
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regulation, where actors formulate their educational policies and submit them to 
their members. 
According to loannidou (2007 p. 38), by carrying out an analysis of 


the actors at the transnational level and presenting us with three groups of actors: 


a) International actors - such as UNESCO or OECD 27 , which 
areintergovernmental organizations with no capacity of 
regulationand control over their members, having only the power 
to definean agenda for education they propose, eventually 
assuming arole of great influence in defining educational policies; 
b) Supranational organizations - such as the European Union, 
which have regulatory capacities over their members and also 
have some legal power to shape policies in some areas. The 
European Union does not have explicit power over the education 
of its member states, but it does have an enormous influence 
over the definition of its educational policies; 

c) Non-governmental organizations - such as the European 
Educational Research Association, which operate as_ self- 
organized networks of professionals without interference in 
government policies. 


Thus, according to Novoa (1998, 2005) the practice of regulation in 
the European Union is homogenized or unified by national education systems. 
From this perspective, the regulatory structure imposes, regulates, media, 
compares and identifies common objectives to disseminate the results achieved. 

The instruments, mechanisms and tools used by the EU to achieve 
regulation are those described as: International Actors (OECD and UNESCO); 
Supranational Organizations and Non-Governmental Organizations. Neves 
(2018) highlights that each one plays the role of monitoring, measuring and 
comparing educational outcomes, the evaluation of outcomes and the 
performance of education systems, thus highlighting the political regulation of 
education. 

We find in Barroso (2003) the argument that the process of regulation 
is not restricted to the imposition of standards, as the EU does, but in logical 
differences of interpretation, resulting in confrontations and negotiation exercised 
by individual international actors (OECD and UNESCO) and formal (Professional 
Class Organs) and informal (Unions, Associations, etc.) structures, adopting 
regulation from a perspective of social regulation. 


27 Organization for Economic Cooperation and Development is an intergovernmental economic 
organization with 37 member countries, founded in 1961 to stimulate economic progress and world trade. 
http://www.oecd.or. 
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5.2 THE EDUCATIONAL SYSTEM IN PORTUGAL 


Published in Eurydice (2019) the key characteristics of the 
Portuguese education system: 

The Ministry of Education (ME) is responsible for general non higher 
education in Portugal. The channels profissionalizantes have double 
guardianship of the ME and the Ministry of Labour, Solidarity and Social Security 
(MTSSS). Higher Education is the responsibility of the Ministry of Science, 
Technology and Higher Education (MCTES), which is also responsible for 
definigao and the application of policies to the Cientifico system and national 
technology. Adult Education and Training is the joint responsibility of the ME and 
the MTSSS. 

The assignments of these three ministries, according to Eurydice 
(2019), are pursued through services integrated in the direct administration of the 
State, bodies integrated in the indirect administration of the State, consultative 
bodies, and other structures and entities belonging to the business sector of the 
State. The school network is organized into School Groups that integrate 
educational establishments that provide all levels of education from Pre-School 
to High School. 

According to Euridice (2019), the Portuguese Educational System?® 
is very centralized both at the organization level and at financiamento. However, 
Pre-School, Elementary and Secondary schools enjoy some autonomy, namely 
at the pedagogical level, the management of teaching schedules and non- 
teaching staff. Higher Education institutions enjoy a high level of autonomy. 

The compulsory schooling 29 described in Eurydice (2019) lasts 
for12 years, starting at the age of 6 and ending at the age of 18 or with the 
completion of secondary education (ISCED 3). Public education is free and 
universal. 

There is only one career profissional on Eurydice (2019) for teachers 
of all levels of non higher education, from Pre-School Education to Secondary 


28 SEP: Cite1.Preschool Education; Cite2. Basic Education; Cite3. Secondary Education; Cite4. Post 
Secondary Education; Cite5. Higher Education; Cite6. Undergraduate; Cite7. Master's Degree; Cite8. PhD. 
https://eacea.ec.europa.eu/national-policies/eurydice/content/higher-education-60 pt-pt 

2° Structure of European educational systems Pre-school compulsory education (Cycle 1, 2 and 3) Cite1; 
Cite2 and Cite 3. https://eacea.ec.europa.eu/national-policies/eurydice/content/structure-european- 
education-systems-201920-schematic-diagrams en 
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Education, and the entry into the career requires a second cycle diploma‘? 
(Masters). In the Autonomous Regions of the Azores and Madeira, the 
administration of education is the responsibility of the Regional Governments, 
through their respective Regional Education Departments, which adapt the 
national education policy to a regional plan and the human, material and 


financeiros resources. 


The Portuguese education system described by Eurydice (2019) is 
divided into Pre-school Education (from 3 years old until entering Basic 
Education), Basic Education (from 6 to 15 years old) and Secondary Education 
(from 15 to 18 years old). The attendance of the Pre-school Education is optional, 
recognizing to the family the first role in the education of filhos, consecrating 
however, its universality for all the children from the year in which they reach 4 
years of age (Law No. 85/ 2009 of August 27, amended by Law No. 65/2015 of 
July 3). 


This level of education and training includes different types of courses (Decree- 
Law No 139/2012 of June 5): 


Percursos de Educacao e Formacao Anos de Escolaridade 


Cursos Cientifico-Humenisticos 

Cursos Profissiona's 

Cursos Artisticos Especializados 

Cursos com planes proorios (Cursos Cientifico-Tecnologicos) | 10.2, 11.¢, 12.2 
Cursas de Fnsino e Formagao 

Cursos de Aprendizagem 


Ensino Secundario na modalidade de Ensino Recorrente Adultes 





Box 10 - Different types of training in Education in Portugal 
Source: https://eacea.ec.europa.eu/national-policies/eurydice/content/portugal_en 





The Bologna Process owes its name to the so-called Bologna Declaration, which was signed on June 


19,1999, in the city of Bologna (Italy), by the ministers responsible for higher education from 29 

European countries, including Portugal. It is a process of intergovernmental reform at the European level 
that is partofthe objectives of the Lisbon Strategy and aims to achieve the European Higher Education 
Area. It createscycles - 2nd Master Cycle and 3rd Doctorate Cycle. https://www.uc.pt/candidatos- 
internacionais/sistema_graus/processo-bolonha. 
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Also according to Eurydice (2019) the Cientifico-humanistic Courses 
are oriented to the continuation of studies in Higher Education; the Profissionais 
Courses are oriented to the qualificagao profissional of the students in order to 
enter the active life, also allowing them the access to the continuation of studies 
conferring double certificagao; The Specialized Artistic Courses are vocationed, 
depending on the artistic area, for the continuation of studies (music) or oriented 
in the double perspective of the insertion in the world of work and the continuation 
of studies (visual arts, audiovisual and dance); The Courses with Own Plans are 
courses with plans of study offered by some private schools and subject to 
approval by the competent service of the Ministry of Education. 


5.2.1 Higher Education in Portugal 


According to Eurydice (2019), Higher Education is structured in 
accordance with the Bologna principles and aims to ensure a solid cientifica and 
cultural preparation; provide technical training that enables the exercise of 
activities profissionais and cultural and fosters the development of capabilities for 
design, innovation and critical analysis. 

Also according to Eurydice (2019), Higher Education is organized in 
a binary system, consisting of university education and polytechnic education, 
and university education should be oriented towards the provision of solid training 
cientificas, joining efforts and skills of teaching and research units and polytechnic 
education focus especially on vocational training and advanced technical 
training, oriented profissionalmente. 

Also described by Eurydice (2019) university education is given at 
universities, university institutes and other university education institutions and 
polytechnic education at polytechnic institutes and other polytechnic education 
institutions, both of which may be provided by public, private or cooperative 
institutions. Despite this binary organization, there are some _ polytechnic 
institutions integrated in universities. 

A National Credit System for Education and Training Profissionais 
was also implemented at the beginning of 2017, which allows credit points to be 
assigned to the qualificagdes integrated in the National Catalogue of 
Qualificagdes (CNQ) and to other training courses certificadas, provided that they 


50 


are registered in the Information and Management System for Educational and 
Training Offer (SIGO) and meet the quality assurance criteria in force. This 
system incorporates the principles of the European Credit System for Education 
and Training Profissional (ECVET), favoring mobility within the European space. 

According to Eurydice (2019), for a more in-depth description of 
these and other topics concerning the national education system, please consult: 
Education Funding; Pre-school Education; Basic Education; Secondary 
Education; Higher Education; Adult Education and Training; Teachers and Non- 
Teaching Staff; Educational Support and Guidance; Mobility and 
Internationalization. 


Age of students Programme duration (years| 
2 2 3 ft & £& FT Ee oe 0 4 a As oe ee! 2 e£&£ as £ Si @ 7 4 
_ 
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Jardim de Ensino basico Ensino basico Ensino secundério 


infancia Cursos cientifico-humanisticos 





Ensino universitario 


Cursos artisticos especializados 


Ensino politécnico 
Cursos profissionais 


Cursos de especializagdo tecnologica 


6 Early childhood education and care (for which the Ministry of Education is not responsible) 1B) Secondary vocational education 
fal Early chikdhood education and care (for which the Ministry of Education is responsible) B Post-secondary non-tertiary education 
p [BD Primary education BB Single structure [1] Secondary general education [Bj Tertiary education (full-time) 
ortugal 
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Compulsory part-time education [>> Study abroad — -/n/- Compulsory work experience + its duration phased out during (year) 


Figure 5 - Structure of the National Education System of Portugal 
Source: https://eacea.ec.europa.eu/national-policies/eurydice/content/portugal en 


Figure 5 does not describe the bodies that are part of the educational 
community: National Student Fee and Support Systems (the national tuition and 
support systems (grants and loans) for students in Higher Education in Europe); 
Organisation of the Academic Year in Higher Education; Organisation of School 
Time in Europe (Primary and General Secondary Education); 
Recommended Annual Instruction Time in Full-Time Compulsory Education in 
Europe. 
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5.3 THE ASSESSMENT SYSTEM OF PORTUGAL 


The Portuguese Evaluation System is supported by the Instituto de 
Avaliacéo Educativa’. (IAVE) I.P, created by Decree-Law 102/2013 that 
succeeded the Gabinete de Avaliagao Educacional (GAVE). IAVE’s governing 
bodies are the Board of Directors, the General Council, the Scientific Council and 
the Single Auditor. As a partner of the Ministry of Education, IAVE’s main purpose 
is to promote the continuous improvement of the educational system, contributing 
to its quality, effectiveness and efficiency. It is focused on the design, application 


and classification of evaluation instruments and on the processing and sharing of 


information generated by the analysis of evaluation results. 


[AVE ©»: 





Figure 6 - Organization Chart of the Structure of the Institute for Educational Evaluation, I.P. 
Source: http://www. iave.pt/index.php/iave/quem-somos 


The Institute of Educational Evaluation, |.P. operates the international 
studies of evaluation being responsible for the application, in Portugal, of the 
evaluations of students who participate in the international studies PISA, TIMSS 
4th year, TIMSS Advanced, PIRLS, ePIRLS and ICILS. 

Estes exames internacionais relacionados sao: PISA - O Programme for 
International Student Assessment; TIMSS - Trends in International Mathematics 
and Science Study 4.° ano; TIMSS - Trends in International Mathematics and 
Science Study Advanced, PIRLS - Progress in International Reading Literacy; 
ePIRLS - Progress in International Reading Literacy e ICILS - International 
Computer and Information Literacy Study. 





31 IAVE is governed by compliance with the requirements and technical specifications stipulated by the 
Organization for Economic Cooperation and Development (OECD) and the International Association for 
the Evaluation of Educational Achievement (IEA). http://www.iave.pt/ 
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5.3.1 Progress in International Reading Literacy Study - PIRLS 


PIRLS - Progress in International Reading Literacy Study, free 
translation: Progress in International Reading Literacy Study, created by the 
Association for the Evaluation of Educational Achievement (IEA) evaluates the 
reading literacy of 4th grade students. 


The study is developed by IEA (2019) and was first applied in 2001. 
Portugal has been participating in this study since 2011. PIRLS’ evaluation of 
Reading Literacy is based on a comprehensive notion of what it is to know how 
to read and its Framework (Quadro de Referéncia) was designed to contemplate 
purposes and processes of understanding Reading that give meaning to the 
concept of Reading Literacy. The purposes contemplate, on one hand, reading 
as a literary experience and, on the other hand, reading as a means to acquire 
and use information. In the analysis of students’ performance, these purposes are 
crossed with the main processes of Reading comprehension: locating and 


removing explicit information; making direct inferences. 
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Figure 7 - Assessment Program 
Source: http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=3418 











i. What are the characteristics of large-scale (i.e., national and international) 
evaluation programs that informed educational policy formulation in developing 
countries? 

ii. How are evaluation program data used to inform educational policy formulation 


in developing countries? 
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iii. At what stages of the policy process are evaluation program data being used 
in developing countries? 

iv. What educational policies in developing countries have resulted from the use 
of evaluation data? 

v. What are the facilitators and barriers to the use of evaluation data in 


educational policy formulation? 


5.3.2 European Survey on Language Competences - ESLC 


European Survey on Language Competences such as ESLC. Free 
translation: European Survey on Language Skills. An initiative of the European 
Commission has as its main objective to assess the foreign language proficiency 
of students, preferably in the final year of ISCED 2 (9th year of schooling), with 
reference to the Common European Framework of Reference for Languages 
(CEFR) **; which designed, applied and analysed the tests to assess foreign 
language proficiency and the questionnaires to the student, the school and the 
teacher. The information in the exam allowed to contextualize the results obtained 
in the test and to add knowledge about the teaching and learning practices of the 
foreign languages evaluated. 

According to Jones, Neil & Gille; Erna & Kortes; Joke & Ashton, 
Karen (2012) the language performance carried out by the ESLC translates the 
linguistic competence into the European framework: 
EN - Holiday photo 





You are on holiday. Send an email to an 
English friend with this photo of your 
holiday. 

Tell your friend about: 

* the hotel 

* the weather 

* what the people are doing 

Write 20-30 words. 

















Figure 8 - Example of writing task in English A1: "vacation photo". 
Source: First European Survey on Language Competences Final Report (2012 p.23). 





32 The Common European Framework of Reference for Languages (CEFR) is an internationally recognized 


standard for describing proficiency in a language. https://www.britishcouncil.org.br/quadro-comum- 
europeu-de-referencia-para-linguas-cefr 
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Illustrate the ESCL task A1 above in English 





Task Achieves A1 Pre-A1 
"Hil I living in Hotel Bellevue and this is nice, | They play voleyball. The namn of the hotel 
At We have swimming pool and a nice resturant. is Belleevue. Have a greates tree. 


. The weather is very good, its sunny and very 
Holiday | 0+. And the people play vollybal! and they are 


photo | nice. 
Good bye!" 


Figure 9 - Example performances for English A1 task "holiday photo". 
Source: First European Survey on Language Competences Final Report (2012 p. 23). 


According to Taylor (2006), the performances used here as examples 
were selected using statistical information about the student’s ability as well as a 
subjective judgment of their overall representativeness of the level. They represent 
a clear achievement or a clear failure, rather than a borderline performance. In 
this report, no explanation is offered as to why a particular example is successful 
or failing, as the objective is simply to illustrate. However, this explanation of the 
criteria for success was important in the ESLC°? , where the examples were 


usedfor training and standardization of markers. 


















M A2 Bi B2 Reading - English 
Type 8 - B2 
Type 6 - B2 
Type 7-B2e 
Type 7-B1 
Type 5-Bie 
-B1 

-A2e 

ype 3-A1 
< easiest ee eee ee ee hardest > 


Figure 10 - Logit scale demonstrating performance and levels. 
Source: First European Survey on Language Competences Final Report (2012 p. 23). 


The scale units are described at the base of figure 9. The horizontal 


axis represents the ability, increasing from left to right on a logit scale**. The 


33 The European Language Skills Survey (ESLC), the first such survey, is designed to collect information on 
the foreign language proficiency of students in the final year of lower secondary education or the second 
year of upper secondary education in the _ participating countries or communities. 
https://op.europa.eu/en/publication-detail/-/publication/42ea89dc-373a-4d4f-aa27- 
9903852cd2e4/language-en/format-PDF/source-119658026 

34Logit as a regression function belongs to the class of statistical models, in which the variable explained 
is, in fact, qualitative, or, being quantitative, it can only observe manifestation of qualitative character 
(OLIVEIRA, 1997). The objective of these models is to reflect a choice between two alternatives, "yes 
orno", "this or that", "exists or does not exist". In the Logit or logistic regression model, Pi is defined as 


, 


the probability can be determined by several factors (explanatory variables). 
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vertical lines are the level cuts, as determined by the default setting, the ends are 


the smallest and largest values. 


5.3.3 Trends in International Mathematics and Science Study - TIMSS 

The Trends in 
(TIMSS). Trends in International Mathematics and Science Study (TIMSS) is an 
international assessment of 4th and 8th grade students’ performance in 


International Mathematics and Science Study 


mathematics and science, also conducted in a digital version and support of the 
study. eTIMSS is developed by the International Association for the Evaluation of 
Educational Achievement (IEA). 

According to Zuzovsky (2006), the model presented in figure 9 fits 


into statistical correlation parameters, to assign a value in mathematics and 


























science. 

Chile (Vv = 6,286, using listwise deletion) (N = 6,269, using listwise deletion) 

1 2 3 1 2 3 
1. “lL enjoy learning math” 1.000 1. “I enjoy learning science” 1.000 
2. “| do wellin math” 0.461 1.000 2. “| do well in science” 0.476 1.000 
3. “learn math quickly” 0.482 0.566 1.000 3. “lI learn science quickly” 0.506 0.538 1.000 
Japan (NV = 4,699, using listwise deletion) (N = 4,781, using listwise deletion) 

1 2 3 1 2 3 
1. “l enjoy learning math” 1.000 1. “| enjoy learning science” 1.000 
2. “ldo well in math” 0.415 1.000 2. “| do well in science” 0.448 ~—- 1.000 
3. “learn math quickly” 0.448 0.541 1.000 3. “I learn science quickly” 0.506 0.560 1.000 
Morocco (N = 2,448, using listwise deletion) (N = 2,527, using listwise deletion) 

1 2 3 1 2 3 
1. “I enjoy learning math” 1.000 1. “l enjoy learning science” 1.000 
2. “I do well in math” 0.400 1.000 2. “| do well in science” 0.319 1.000 
3. “I learn math quickly” 0.448 0.4988 1.000 3. “| learn science quickly” 0.358 0.441 1.000 
United States (V/V =8,592, using listwise deletion) (NV = 8,556, using listwise deletion) 

1 2 3 1 2 3 
1. “l enjoy learning math” 1.000 1. “l enjoy learning science” 1.000 
2. “Ido well in math” 0.514 1.000 2. “| do well in science” 0.563 1.000 
3. “I learn math quickly” 0.518 0.661 1.000 3. “I learn science quickly” 0.581 0.655 1.000 








Figure 11 - Correlations of three measures of perception in selected countries (TIMSS). 
Source: IEA Conference Proceedings (2006, p. 55). 


The parameter used are three possible values of these variables on 
the standard zed scale and were used for these predictions: 1 - minimum value, 
2 - actual average value defined as 0, and 3 - maximum value. Pearson’s 
correlation in the coefficient totaled above r = -.708, p <.001, N = 46. 
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5.3.4 International Computer and Information Literacy Study - ICILS 

International Computer and Information Literacy Study - ICILS. Free translation: 
International Computer and Information Literacy Study. To evaluate digital and 
information literacy in young people between the ages of 13 and 14, and how 
they develop the knowledge and skills within that literacy, fundamental to full 
citizenship in contemporary societies; to know to what extent young people 
around the world are prepared to study, live and work in the digital age. 

According to Fraillon, Schulz & Ayley (2016), the International Study on 
Computer and Information Literacy (ICILS) has studied to what extent young 
people have developed computer and information literacy (CIL) to support their 
ability to participate in the digital age. Many countries recognize the importance 
of information and communication technology (ICT) education to enable citizens 
to develop the skills needed to access information and participate in transactions 
using ICT (European Commission, 2006). We refer to these skills as computer 
and computer literacy, which is defined as "the ability of an individual to use 
computers to investigate, create and communicate in order to participate 
effectively at home, at school, at work and in society" (FRAILLON, SCHULT , & 
AINLEY, 2013, p. 17) 


Antecedents Processes Outcomes 





Wider community Wider community 
Educational system Educational policies 
Availability of ICT internet sources 


School/classroom: School/classroom: 
Characteristics ICT use for teaching/ 
Stated ICT curriculum leaming 

ICT resources CIL/CT Instruction 
eee er ee Ke ee ee 
' 1 ' ' Computer and 

' Student: { ' Student: ; information literacy 
Characteristics ' Learning process ' Computational thinking 


oe ee 


i 
Home environment: i 1 Home environment: 
Family background ‘ ' ICT use at home 
and ICT resources 


pee eee beet ee oe ee eee 








Figure 12 - CILS Structure 
Source: https://doi.org/1 0.1007/978-3-030-19389-8 
5.3.5 Programme for International Student Assessment - PISA 


The Programme for International Student Assessment - PISA, free 
translation: International Student Assessment Programme, developed by the 


OECD, aims to assess whether students aged 15 - the age at which, in most 
participating countries, students are at the end of compulsory schooling - are 


prepared to face the challenges of daily life. 
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OECD (2009) describes that PISA is designed to assess whether 
students are able to mobilize their Reading, Mathematics or Science skills in 
solving day-to-day situations, and also whether they are able to reproduce the 
knowledge acquired in these areas. The assessment is part of the Collaborative 
Problem Solving and Financial Literacy. 


According to Ananiadou, K. & Claro, M. (2009), PISA exams are 
prepared according to a PISA Framework, common to all OECD countries. PISA 
applications occur in three-year cycles. In each PISA cycle, emphasis is given to 


one of the areas evaluated Reading, Mathematics or Science. 


Pensamento e acao matematica 
Conceitos matematicos, conhecimentos e habilidades. 


Capacidades fundamentais da matemiatica: comunicar; “matematizar”; representar; 
raciocinar e argumentar; delinear estratégias para a solucao de problemas; utilizar 
linguagem simbolica; formal e técnica e fazer operacdes; usar ferramentas matemiaticas. 


Processos: formular, empregar, interpretar/avaliar. 


Problema no contexto Formular Problema matematico 


Empregar 


Avaliar 


Resultados no contexto Resultados matematicos 


Figure 13 - Model of mathematical literacy in practice. 
Source: OCDE (2019), PISA 2018 Assessment and Analytical Framework. 





Still according to OECD (2018), in its PISA reading literacy scale it 
was divided into several levels of proficiency. Seven of these levels - 1b, 1a, 2, 
3, 4, 5 and 6, in increasing order of proficiency, were used to describe reading 
proficiency on the PISA 2009, 2012 and 2015 tests. We found in the 
score/positioning a limit between the unchanged Reading proficiency levels, the 
descriptions for use of the levels were remodeled to reflect a new Reading 
situation that was evaluated for the first time in 2018. For example, levels 3, 4, 5 
and 6 as defined on PISA 2018. 
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Processamento de texto 


§ 
3 
cS 
5 


Localizar inforenagdes 
- Acessar @ recuperar informacdes dentro do texto 
- Buscar e selecionar o texto relevante 


Compreensjo 
+ Representar o sentido literal 
- Integrar e¢ gerar inferéncias 


Avaliar e refletir 
- Avallat qualidade e credibilidade 
- Refletir sobre o conteudo e 3 forma 
- Detectar e lidar com conflitos 





Figure 14 - Reading matrix processes used in PISA 2018. 
Source: OCDE (2019), PISA 2018 Assessment and Analytical Framework. 


It is also apparent from the OECD report (2018) that the distribution 


of questions for the assessment of literacy in Reading of 2018 was changed and 


contrasted with the distribution of questions from the 2015 assessment. 





oe 2015 MATRIZ 2018 
TEXTOS MULTIPLOS 
TEXTO UNICO 65% 

35% 

Acessar e recuperar Localizar informacgées , = Buscar e selecionar 
Pesq Hi 
25% 25% SERS CMe tee texto relevante 10% 
. > Representar o significado literal 

Integrar e interpretar Compreender 15% Inteqrar e gerar 
50% 45% inferéncias 15% 


Integrar e gerar inferéncias 15% 


Avaliar a qualidade e credibilidade 


Refletir e avaliar 25% 


Avaliar e refletir e Corroborar e lidar com 
30% Refietir sobre o conteddo e a conflito 10% 
forma 20 % 


Figure 15 - Distribution of questions per matrix. 
Source: Fonte: OCDE (2019), PISA 2018 Assessment and Analytical Framework. 


To define the matrix of the scientific content of the OECD (2019a) we 


use three variables composed of the parameters: Explain, Evaluate and Interpret 


to quantify the questions of the exam. 
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Letramento cientifico 6 a capacidade de se envolver com questGes relacionadas 
com a ciéncia e com a ideia da ciéncia, como cidadao reflexivo. Uma pessoa letrada 
cientificamente, portanto, esta disposta a participar de discussao fundamentada 
sobre ciéncia e tecnologia, o que exige as competéncias de: 


1. explicar fenédmenos cientificamente: reconhecer, oferecer e avaliar explicacdes 
para uma gama de fenédmenos naturais e tecnoldgicos; 


2. avaliar e planejar investigac6es cientificas: descrever e avaliar investigagdes 
cientificas e propor formas de abordar questées cientificamente; 


3. interpretar dados e evidéncias cientificamente: analisar e avaliar dados, 
afirmag6es e argumentos em uma variedade de representac6es, e tirar conclusdes 
cientificas apropriadas. 


Figure 16 - Presents the definition of scientific literacy used in PISA. 
Source: OCDE (2019), PISA 2018 Assessment and Analytical Framework. 


The three interrelations of the dimension of the scientific literacy of 
Pisa are also presented* in figure 14. 







Competéncias 





















Exi ue ; 
os individuos « Explicar.fenOmenos A demonstracao das et 
demonstrem cape competéncias depende de * Conceitos/conteddos 
+ Avaliar e planejar (de ciéncias) 
experimentos cientificos + Procedimental e 
+ Interpretar dados e epistemologico (sobre 
evidéncias cientificamente ciéncias) 


Figure 17 - The interrelationships between the dimensions of scientific literacy 
Source: OCDE (2019), PISA 2018 Assessment and Analytical Framework 


Still according to OECD (2019), the dimensions foreseen for the exam, 
are framed in levels of cognitive demand of "deepening" and "knowledge", to 
undertake the student in a mental process necessary to solve the issue. The use of 
a cognitive demand raises the level of difficulty employed. The exam concentrates 
a format that makes it difficult for the student to answer and become familiar with latent 
scientific questions. Comparatively we have: the number and the degree of 
complexity involved in knowledge. The level of familiarity of the 
student with the knowledge. 


35 PISA is the OECD Program for International Student Assessment. PISA measures the ability of 15-year- olds 
to use their knowledge and skills in reading, math and science to meet real-life challenges. 


https://www.oecd.org/pisa/aboutpisa/ 
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6 ASSESSMENT IN THE UNITED STATES 


In this Chapter, we will describe how evaluation in the United States of 
America evolved through authors such as: Horace MANN (1845), Joseph RICE 
(1898), Ralph TYLER (1945) considered the father of modern educational evaluation 
for Benjamin S. BLOOM (1950) who coined the cognitive domain in the teaching- 
learning process, to Michael SCRIVEN (1960), Lee Joseph CRONBACH (1963) who 
created the methodology in evaluation, Robert STAKE (1967), and Daniel 
STUFFEBEAM (1971) who influenced the development of educational evaluation as 
an integral part of the teaching and learning process to be a quality and aggregating 
factor, and how the contemporary educational evaluation process is rooted in the 
evaluation and measurement of the student (Internationalization of evaluative 


metrics). 


6.1 TYPES OF ASSESSMENT PRACTICED IN THE UNITED STATES 

Among the various assessment practiced in the United States in the 
NCES Educational Framework*® we have: PISA, TIMSS, CCSS - Common 
Core State Standards; FAIR TEST- Model Assessement Report; NCLB - No Child Left 
Behind; ESSA - Every Student Succeeds Arts; ECERS - Early Childhood Environment 
Rating Scale among others, existing in the USA, which will be scored in this Chapter. 


We continue to describe the types of structured assessments in the 
United States, the NCES conducts several international assessments and surveys 
in the United States, this includes: the Study of Trends in Mathematics and 
International Science (TIMSS), the Study of Progress in International Literacy in 
Reading (PIRLS) and the International Study of Literacy in Information Technology 
(ICILS); Evaluations are organized by the International Association for Educational 
Performance Evaluation (IEA) the International Student Evaluation Program 
(PISA); the International Adult Competency Evaluation Program (PIAAC); the 
International Teaching and Learning Survey (TALIS); and the International Initial 
Learning Survey (IELS), which are organized by the OECD. 





36 structure of education in the USA. Source: NCES - National Center for Education Statistics: 
https://www.researchgate.net/figure/Figura-3-Estrutura-de-educacao-nos-EUA-Fonte-NCES-National- 
Center-for-Education_fig3_ 28740941 
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We also have standardized and structured Tests/Assessment: 





STANDARDIZED 
TESTS 


TESTS FOR PRE-SCHOOL STUDENTS 


HIGHER EDUCATION 
TEST ADMISSION 





ACT - A multiple-choice 
curriculum-based 
assessment that tests 


reading, English, 
mathematics and 
science, with an 


optional essay section. 


Common Core State Testing. 


GRE: a standardized test of 
verbal reasoning, quantitative 
reasoning and analytical writing 
that measures readiness for 
graduate studies. 





Newfoundland - is a_ series of student 


DAT: a multiple-choice test to 





SAT - atest that performance tests produced by McGraw-Hill to | measure general academic 
measures critical assess reading comprehension, language arts, | ability, understanding of 
reading, writing and mathematics, science, social __ studies, | scientific information and 
math skills. vocabulary, ortografia, and other areas of| perceptive ability, used for 
knowledge. admission to dental schools. 
GPA (Grade _ Point | International Baccalaureate Exam - LSAT: a standardized test that 
Average) "grade | International Baccalaureate Exam measures the acquired reading 


average" and is widely 
used in the United 
States. Some _ school 
evaluation systems. 


and verbal reasoning skills that 


law schools use as an 
assessment factor for 
admission. 





NAEP - National 
Evaluation of 
Educational Progress. 
Evaluations are given 
more frequently in 
math, reading, science 
and writing 


Stanford Achievement Test - Standford 
Performance Test: Often called SAT 10, this 
test produced by educational publisher Pearson 
is used here and abroad to evaluate skills in 
reading comprehension, math, problem solving, 
language, ortografia, listening comprehension, 
science and social sciences. 


GMAT: a standardized test for 
MBA candidates that measures 
the basic skills of verbal writing, 
mathematics, and _ analytics 
that have been developed over 
time through education and 
work. 





DBA - New digitally 
based evaluations. 


STAR Tests - STAR tests were created by 
Renaissance Learning for use in elementary 
and high school. The tests are unique because 
they are completed on the computer and use 
adaptive technology. Educators can use the 
tests to evaluate students in reading, early 
literacy or math, monitoring students’ progress 
and preparing them for more complex tests. 


MCAT: a standardized 
multiple-choice exam that 
evaluates problem solving, 
critical thinking, writing skills 
and knowledge of scientific 
concepts and principles as a 
prerequisite for the study of 
medicine. 





TEL - Technology and 
Engineering of Literacy 
(Assessment). 


PARCC and SBAC_ Tests: PARCC 
(Partnership for College and Career Readiness 
Assessment) and SBAC (Smarter Balanced 
Assessment Consortium) 





New York’ Performance — Standards 
Consortium Test: is a coalition of New York 
State public high schools that has developed a 
proven, student-focused performance 
evaluation system developed by profissionais 
for its 388 member schools in the city and state 
of New York. 








Learning Record and Work Sampling 
System. 








Box 11 - Standardized American Tests. 
Source: Prepared for this work. 


The National Center for Education Statistics - NCES EUDE is part of the Institute 


of Educational Sciences - IES, responsible for collecting, analyzing and publishing statistics 


on education and public school finances in the United States and also applying international 


assessments. Among the evaluations analyzed, the following are highlighted in Box 11. 
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Abbreviation Early Childhood Abbreviation Educational Assessment Surveys 
Education Research 
ECLS Early Childhood Longitudinal Study NAEP National Assessment of Educational 
Progress 
Research in Secondary NALS National Adult Literacy 
Elementary Education Search 
CDD Main Database in Foundational and NAAL National Adult Assessment 
High Literacy 
PSS Research of the Universe of Private TIMSS Trends in International 
Schools Study of Mathematics and Sciences 
TPS Teacher Follow-up Survey PISA International Student Assessment Program 
SASS Teacher Follow-up Survey TALS International Adult Literacy 
Search 
NLS:72 National Longitudinal Study High School ALL Adult literacy and life skills Research 
HS&B Longitudinal Study High School and PIRLS Progress in International 
Beyond Study of reading literacy 
NLS:88 Longitudinal Study of National Domestic Surveys 
ELS:2002 Educational Longitudinal Study - 2002 NHES National House Educational Research 
Program 
Post-secondary and CPS Current Educational Population Survey 
Adult 
IPEDS Post-secondary integrated to the sscs Crime and Security Research at School 
Educational 
NSOPF National Study of Higher Education Staff HST High School Transcription Studies 
NPSAS National Higher Education ; 
Student Assistance Study Library Search 
BPS Beginning of post-secondary SLS School and Media Library Center 
Longitudinal 
Student Study 
B&B Bachelor’s Degree and Beyond ALS [Academic Library Research 
SED Doctorate Researches Obtained 














Box 12 - Evaluations analyzed by NCES. 





Source: Prepared for this work. 


The statistical models used by the NCES, are contained in the Handbook of Survey 
Methods in order to provide standards and techniques of statistics to the user of relevant 
information on National Education. In addition, the focus of the methodology of the National 
Center for Education Statistics and present a report in which provides a summary description 
of data collected from each program. 


a NCES 
| 

National Center for 
Education Statistics 





Figure 18 - Educational Data Analyzed by NCES. 
Source: NCES Handbook of Survey Methods 
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OFFICE OF THE COMMISSIONER 


STATISTICAL STANDARDS AND ANNUAL REPORTS AND 
DATA CONFIDENTIALITY STAFF INFORMATION STAFF 








ASSESSMENTS SAMPLE SURVEYS ADMINISTRATIVE DATA 
DIVISION DIVISION DIVISION 


MAPPING 


REPORTING AND LONGITUDINAL SURVEYS STATE LONGITUDINAL 
DISSEMINATION BRANCH BRANCH DATA SYSTEMS BRANCH 


ADMINISTRATIVE SUPPORT 
SERVICES BRANCH HSLS/ELS/NELS/HS&B 


BPS/B&B 


NATIONAL ASSESSMENT POSTSECONDARY BRANCH 
BRANCH 


CROSS SECTIONAL ELEMENTARY/SECONDARY 
SURVEYS BRANCH ARANCH 


Ras 


INTERNATIONAL 
ASSESSMENT BRANCH 





TFS/PFS/BTLS ccD 


PIRLS 


sf 


TIMSS 


SS50CS/SCS 
PIAAC 4s 


FRSS/PEQIS 
TAUS “hi 


Figure 19 - Structure of the Assessment programs analyzed by NCES. 
Source: NCES Handbook of Survey Methods. 
Analyzing the 6 (six) areas evaluated by NCES: Early 
Childhood Education Research; Secondary Elementary Education Research; 
Post-secondary and Adult [Education Research; [Educational Evaluation 
Research; Domestic Research and Library Research. These programs collect, 
treat, analyze and group the data, which will be presented through information. 
The study of this thesis is linked to the international evaluations, their metrics, 


modeling and values presented, quantifying a ranking among the countries. 
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6.2 THIS - ANEW NATIONAL ASSESSMENT IN THE USA 


Created by the knowledgeworks and launched by the Department 
of Education during President Obama’s term in 2017 as a new national evaluation 
proposal, the ESSA - Every Student Succeeds Act, a free translation: "Law that 
All Students Should Succeed"; initially it was applied in 7 (seven) states. It was 
created as a state and district policy framework to support next generation 


evaluation systems and a personalized education. 





Figure 20 - Application map of ESSA 2018. 
Source: A Supporting State Policy Framework for Next Generation Assessment Systems. 


Also designed as an innovative assessment system, ESSA allows 
students to conduct the assessment when they are ready, thus providing the 
opportunity to demonstrate mastery of the content several times. States and 
districts that adopt ESSA use this flexibility to build an educational system with 
assessment that meets the needs of each student, not to perpetuate ineffective 
test preparation strategies. This feature allows the State or District to offer 
assessments at different times of the year, allowing for multiple assessment 
periods. Thus, districts measure competencies as students move toward 
proficiency, while aligning assessments to students’ own interests and goals; 
providing learning experiences outside of the school building and, in some cases, 


outside traditional school hours, with exemptions from time requirements. 


ESSA is also characterized by creating significant flexibility to 
design customized education systems that meet the needs of each student, as 


shown in the infographic of figure 20. 
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ESSA and Personalized Learning Dashboard 


39 state ESSA plans reference personalized learning. Here’s a snapshot of how states are advancing this approach. 
















VISION 


17 states 
worporate personalized 
learning into thew vision 
tor ESSA Impiemematon. 


EXCELLENT 
EDUCATORS 


10 states 

will explore he develoamerm ofa 
statewide system of Micro-credentials 
for educators to personalize and enhance 
professional development. 


14 states 
will hunch an oniine professional devesopment pintiorm tor 
educators to make personalized professional 
development evaianic eaytime, anywhere. 


it ee 





















ACCOUNTABILITY SCHOOL 


IMPROVEMENT 
19 states 


wi esinbisn dashboards 
to peovkie parents, students, and other 
education stnkehokiers with comarenensive data 
on school quailty and student oppertunities. 


11 states 


wi prontze personalized learning 
Strategles tor supporting schools Mentnes 
for improwement 


wit prontze strategies for Meaningful 
community engagement in the cevelonment 
amd implementabon of school impeowement plans. 


25 states 


w@ use an academic proficiency indicator thet accounts 
for the performance level of each stucent in a school, 
suchas Scale scores or performance 
Indices, incentwzing impraved performance for 
each student recardiess of whether they 
score Defow of above proficient. 






SUPPORTING 
ALL STUDENTS 


35 states 

WE Incorporate a high schoo! Indicator that 
emphasizes Multiple pathways 
for demonstrating cofeoe and 

Pace nome: Bl 


21 states a “=e 
wi prontize the use of Tite Iv . " 
funcs to support schools anc disincts to 

cotter social-emotional learning and 
whole child supports to ermance or 


imprave existing course offerings. 


& 


37 states 


wi Incorporate an extended-year 
graduation rate Indicator prowsing crect 


19 states 

wit ensure af stucents nave a personalized 
Jearning plan tnat segns to their academic 
needs, interests. and goals. 


for schools thet successfully graduate students wo 
need adatans time. 





Figure 21 - Infographic ESSA: possibilities of personalized education in 39 States 
Source: https://knowledgeworks.org/resources/essa-personalized-learning-dashboard/ 


Among the strategies established in the ESSA - "Every Student 


Succeeds Act" we 


highlight the main goals of personalized education: 


1]Instruction aligned with rigorous academic standards and 
students with social-emotional skills need to be ready for college, 
career and life; 

[2]Customized instruction that allows each student to design 
learning experiences in line with his or her interests; 

[3]Variable pace of instruction based on individual student needs, 
allowing students to speed up or reserve additional time based 
on their level of mastery; 

[4]Real-time differentiation of instruction, media and interventions 
based on formative data; 

[5]Evaluations and student feedback to ensure that each student 
remains on track for graduation; 

[6]Access to clear and transferable learning objectives and 
assessment results for students and families understand what is 
expected of mastery and advancement (KNOWLEDGEWORKS 
2019. p. 2). 


With these goals established, ESSA is consolidating itself in the USA as the 


main national assessment. 
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7 SURVEY METHODOLOGY 
In this chapter we will describe the structure of the methodology that 
will be used in the research, as well as the procedures necessary for the 
qualitative and quantitative approach used to reach the result. 


7.1 METHODOLOGICAL PATH OF THE RESEARCH 

This research is based on observed phenomena, known facts and 
grounded in data produced. The formulation of hypotheses defined as 
assumptions, the understanding of the evaluation process to promote the 
improvement of teaching-learning is fundamental, aiming the search for a quality 
education, being built based on studies in international evaluations. 

According to Duarte and Furtado (2002, p.15), research is "a rational 
and systematic procedure that aims to provide answers to problems, curiosities 
or challenges that are proposed. Morgam (2007) says that the methodological 
procedures necessary to carry out this study start from the quali-quantitative 
approach that allows describing, analyzing and understanding the variables 
related to the object of study. 

Still according to Neuman (1994), the quali-quantitative method 
approach depends on factors and levels, which can be described as the graphs 
of networks, aggregation patterns, and non-relational networks and computational 
information. Mares (2012) describes that the rapid growth of digital data and 
computationally advanced methods in the social sciences has aggravated, in 
many ways, the tensions between the so-called ‘quantitative’ and 
‘qualitative’ approaches. 

Using the experimental research for the investigation in this study, 
taking into account the high degree of control of the situation of the experiment; 
this type of study is characterized by the direct manipulation of variables related 
to the object of the study "ranking in education rank of the country?” Thus, the 
research, formulates and manipulates factors or variables that refer to the 


phenomenon of research. 





37 OECD -Teaching for the Future: Effective Classroom Practices To Transform Education, Publishing, 
Paris, 2018. https://dx.doi.org/10.1787/9789264293243-en 
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From an objective point of view, the research is characterized as 
experimental. According to Gil (1991), the method provides greater familiarity with 
the problem. In order to make it explicit. For this, it involves the bibliographic 
survey, as well as the study of the Educational Systems of each country and their 
metrics involved for classification. 

After the bibliographic research of the numbers provided by OECD, 
EACEA, NCES and IAVE, we found the concept in Vergara (2004, p. 49) as "the 
study systematized and developed based on material published in books and [...] 
material accessible to the general public"; it started with field work where data 
from the evaluation systems of Portugal (Appendix A), United States (Appendix 
B) and Brazil (Appendix C) were collected and analyzed by modeling. 

Seeking to expand the information on the research methodology, and 
understanding that originally international evaluations were created for economic 
purposes, the Organization for Economic Cooperation and Development (OECD) 
and UNESCO; contemplates as its main proposal to be a forum for discussion on 
issues related to the development and improvement of economic and social 


policies. In the first moment we observe the case in the United States, where 
PISA®® was considered only one of the many studies that evaluate the 
performance of educational systems; thus leading to a very high degree of policy 
convergence. This was not the case in the United States, where PISA was 
considered only one of many studies assessing the performance of educational 
systems. 

In the second moment the process of document analysis of the notes 
released by the OECD, being anchored in the techniques of Content Analysis 
(BARDIN, 2011). At the pre-analysis moment, we perform a floating reading, 
which is thus defined by Minayo (2014, p. 316): 


[...] this moment requires the researcher to take direct and 
intense contact with the field material, allowing himself to be 
impregnated by its contents. The dynamics between the initial 
hypotheses, the emerging hypotheses and the theories related 
to the subject will make the reading progressively more 
suggestive and able to overcome the sensation of chaos. 


38 Highlights of U.S. PISA 2018 Results Web Report (NCES 2020-166 and NCES 2020-072). U.S. Department 
of Education. Institute of Education Sciences, National Center for Education Statistics. Available at 
https://nces.ed.gov/surveys/pisa/pisa2018/index.asp. 
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According to Coutinho (2008); Mcmillan (2000), the qualitative 
research also evidenced in this study, presented through structured procedures 
and able to control and predict phenomena. The use of this type of research is an 
attempt to explain the occurrence of facts that will explain a phenomenon 
(WIERMA, 2000). 

With the proposal of adopting more rigorous criteria in qualitative 
research, Guba & Lincoln (1981, 1988 and 1995) developed on the interpretative/ 
constructivist aegis, what is called "informative quality of data",°° coining terms 
and expressions that demonstrate the interpretative paradigm, that is, words that 


prove to the researcher that the results obtained are true and true, even when it 


comes to subjective interpretation. 











Indicator Translation Meaning (GUBA & LINCOLN) 
Truth Value Own Value From the author’s perspective 
Applicability confirmability | Applicability Ability of other researchers to confirm the 
Reliability constructions of the researcher (Lincoln and Guba, 


1991). 














Consitency Consistency Ability of external researchers to follow the method 
used by the researcher. 

Neutrality Neutrality Characteristic or condition of what is neutral; 
impartiality; exemption. 

Trustworththiness Reliability Action _ satisfactorily, according to certain 
specifications, in a given time interval, under pre- 
established conditions. 

Credibility Credibility Credibility is obtained by "submitting (the results) to 


the approval of the builders of the multiple realities 
under study" (Lincoln and Guba, 1991, p. 296) 





Transferability 


Transferability 


Ability of the study results to be applied in other 
contexts. 





Dependability 


Consistency 


Reliable information, with no changes. 





Negative cases 


Negative Cases 


No peer review of results. 





Triangulation 


Triangulation 


Combine qualitative and quantitative data collection 
methods and sources (interviews, questionnaires, 
observation and field notes, published documents, 
among others). 




















Peer debriefing Peer Review Evaluation of results regarding competence, 
significance and originality conducted by qualified 
specialists who research and submit papers in the 
same area for publication. 

prolonged engagemen Extended An investment in the time deemed necessary to 
achieve the research objectives. 

Involvement 

audit trails Audits Test contradictory information introduced by 

distortions. 





Box 13 - Terms Used in Qualitative Research 


Source: Issues of validity and reliability in qualitative research (COUTINHO, 2008. p.8) 





3° Qualitative research is an umbrella concept that covers various forms of research that helps us 
understand and explain the social phenomenon (MERRIAM, 2002). 
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7.2 DATA TRIANGULATION 


The use of triangulation*® protocols in this study will combine two or 
more points of view from data released by the OECD in international evaluations. 
Having a qualifying approach or quantification method, Flick (1998, p. 228) "if 
there are two opposing points of view or data, the difference is used as a form of 
reflection, a new analysis will be carried out to find an explanation that justifies 
the differences". 

In the third moment we will use the statistical modeling to evidence 
the qualitative and quantitative analysis that involves the link enunciated in 
individual scope (Country) and macro trend (Countries). 

It is common in a survey, involving the quantitative approach, to make 
comparisons of two or more indexes between them, in relation to the variables, 
being thus, the analysis of variance (ANOVA) is a statistical modeling that allows 
decomposing the total variation. Thus, it is possible to compare means between 
them released by the OECD and state, at a certain level of probability, to draw 
conclusions about differences (FERREIRA, 2000; PIMENTEL-GOMES and 
GARCIA, 2002; FERRAO, 2003). 

The methodology for calculating the analysis of variance is related to 
calculating the sum of the total squares and the number of degrees of freedom of 
the experiment. By decomposing the sum of total squares into the sums of 
squares of the various sources of variation considered in the interaction. Using 
the respective degrees of freedom, it is possible to obtain the corresponding 
variances. It then proceeds to a hypothesis __ test(f) to determine whether the 
differences between the variances are significant at a given level of probability 
(FERREIRA, 2000). 

According to the authors Cochran and Cox (1981) the validity of variance 
analysis is linked to four factors: (1) the various effects included in the statistical 
model are additives; (2) the errors are independent from observation to 


2 


observation; (3) all errors have the same variance “ ; and (4) the errors have 


normal distribution. 





40 Interest in triangulation for research is not a recent phenomenon. From the conception of the idea in 
the 50s with the studies in psychology by Campbell & Fiskie (1959) through the idea of multiple 
operationism, until its popularization in the 80s (Bazeley, 2002), triangulation is still today a subject of 
great debate and discussion (WEBB, CAMPBELL, SCHWARTZ, & SECHREST, 1966). 
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Still on statistical modeling, we will also make use of the tool known 
as test of averages, which makes multiple comparisons of averages, when these 
have qualitative characteristics. Complementing the F test of variance analysis, 
the comparison test will show if there is significance between the comparison in 
at least two means (FERRAO, 2003). 

Cardellino and Siewerdt (1992) point out that the mean comparison 
test, if not used correctly, can impair the conclusion from the results presented. 
Emphasizing that the statistical methodology used in the description of the F test 
can restrict algebraic and computational aspects, using statistical procedures 
appropriate to the types of data (SON and LUCIO, 2008) so we have: 


Orthogonal contrasts *' 
Be x a linear combination of the true averages of a trial with | 
treatment given by: 


X = Crm + Come + ...+ Crm=Xi_, C1 m; (1) 


where Ci is the coefficient applied to the mean of treatment ir miis the mean 
of treatment i. 


One can say that X is a constant if the sum of the C coefficients of 
the linear combination is equal to zero. 


Be now two contrasts X1 and X2 given by: 


X1= Yj, C1, m; e Xo= Yi, C2; m; (2) 
X1 and X2 are said to be orthogonal if the sum of the products of the 
corresponding coefficients in the two contrasts is zero, i.e. if 


X1= Yin C1; C2; = 0 (3) 


Following the same reasoning, a contrast is said to be orthogonal if 
all the contrast pairs of the set are orthogonal (VIEIRA and HOFFMANN, 1989). 





41 The technique of orthogonal contrasts with a degree of freedom is simple and very efficient in the 
analysis of experimental data, for example, in obtaining main effects, interaction effects and nested 
effects, comparisons between groups of means (our griffon) and in obtaining specific residues. Nogueira 
(2004). https://www.scielo.br/pdf/sa/v61n1/a20v61n1.pdf 
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From a practical point of view, orthogonality indicates that the 
variation of one contrast is entirely independent of the variation of the other 
contrast whatever is orthogonal to it (SON and LUCIO, 2008). 


Given a contrast X, the estimate of this contrast is obtained by: 


X =D Cm (4) 


where 77; is the estimate of the average treatment i. 


The estimated contrast variance is given by: 


() = Ma [2-2 (“ /n, )] (5) 


in which QMR is the mean square of the residue of the analysis 


of variance and ni is the number of repetitions of the treatment. 


Therefore, the sum of the squares of the estimate of a contrast X has 


a degree of freedom, and is calculated by: 


SQ(x)-—* 


Xiea(“/ n) (6) 


The value of F is calculated as follows: 


>, 5Q(X) WHO ; 


In which GLR is the number of degrees of freedom of the residue. 


Given an experiment where orthogonal contrast was formulated (r =I -1), it was 
decided to reject a certain situation HO: Xr=0 se F(X,)>F@(1:GLr). 


In this sense, it is concluded that the effect defined by the contrast is 
significant in level of error probability (STORCK et al., 2000). 
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The Tukey _ test will also be used in this study, which requires an 
equal number of repetitions in all treatments, serving to present any contrast 
between two averages, not requiring these parameters to be orthogonal. In the 
Turkey test the minimum significant difference (dms) is calculated for comparison 


between the means presented (HAIR et al., 2009). 


The minimum significant difference (dms) for an equal number of 


repetitions in all treatments is calculated by: 


QMp 


dms = qa (I; GLR)J n> (8) 


where q (I;GLR) is the total range studied in level _ of error for | treatment and 


GLR degrees of freedom of the residue, and n is the repetition number. 


When the number of repetitions between two treatments i and i’ is 
different, Hair et al. (2009) recommend the following formula to calculate the 


minimum significant difference: 
[ome (@ —) 
dms = qa (I; GLrp)y 2 \u +” (9) 


where ni and ni’ are the numbers of repetitions in treatments i and i’, respectively. 





The hypotheses to be tested are the following: 


HO : m= mj (10) 
H1 mi mj (11) 


The conclusions obtained with Tukey’s test are: 


To finish the test Tukey works with fully significant difference - WSD. 


Tukey’s strategy consists in defining the smallest significant difference. 


Also as a tool in statistical modeling to be used in this research, the 
Sheffé Test is recommended to ascertain stipulated contrasts, after 
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previous analysis of the data (CARDELLINO and SIEWERDT, 1992). The Sheffé 
Test is very versatile, being practiced when the number of repetitions per 
treatment is different and when the contrasts have more than two means. We can 
highlight as disadvantages of the test its reduced power according to Vieira and 
Hoffman (1989). 

According to Hair et al., (2009) the Sheffé test is more rigorous than Tukey’s 
test, and is more used to test more complex contrasts, and it is inadvisable to use 


it to compare averages two by two. 


dms = (U — 1)0(%)Fa(GL rp -GLR) (12) 


where F (GLTR; GLR) is the tabulated value of F in level error for GLTR 
degrees of freedom of treatment and GLR degrees of freedom of waste, | is the 


number of treatments, and Y (*) is the estimated variance of contrast given by: 


V(X)= onl (*/ my) a 


The hypotheses are: 


HO : C = 0 ( Similar treatment) 
H1:C #0 (different treatment) 


The conclusions obtained with the Scheffé test are: 
If |C] => S HO is rejected 
If |C| <S: HO is accepted 


To finish the tests that will be used in this thesis, we present the 
Student Test, being recognized to test two or more contrasts that involve groups 
of media Hair et al., (2009). Having its applicability in orthogonal contrasts, 
established before the data are examined, i.e., before the comparisons made, 
should be at most equal to the number of degrees of freedom of treatment. The 


test equation is the following: 
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C-0 


2" 2 

QMR (2 +2 ..4+“k) 
f=y (14) 
The hypotheses are: 

HO : C = 0 (similar treatment) 


H1:C #0 (different treatment) 


The Conclusions obtained with the "t" test are the following: 
If |t- | = t, (degree of freedom of the waste) : HO is rejected 
C| it 
If | | < “ (degree of freedom of the residue) : HO accepted 
7. 2.1 Analysis of Variance - ANOVA 
In this experiment we will have groups of averages, which will be 
subdivided in the design, with | levels of factor A (divided), K levels of factor B 


(subgroups of averages) and J repetitions, the sum of the squares can be 
obtained by the following formulas: 


1 
SGa- jx yy; DKVijx)2¢ (15) 
1 
SQa= x Lij(LeVijx)2-C —~ SQa (16) 
1 
SQu-7 YK(Di ViVijx)2 e (17) 
I 
SQas =] Lik (Lj%ije)2—¢ —SQa-SQp (18) 
SQrotat = Dijk Yijx2-C (19) 
SQr (B) = AQToTAL— SQrRiA)— SQA— SQB— SQaB (20) 
Dijk y;; 
| Suk tik (21) 
The Value of Constant C is Calculated by: C=\ yx /4 


Also to perform the variance analysis it is necessary to calculate the 
sums of total square, residue or independent of regression, by means of the 
following modeling: 


Sum of Total Square ( SQTotal ) 
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SQTotal = Lyi (=) (22) 


Sum of Squares of Regression (SQREG) 


ret —_ = is ‘ Rs 3 a 2 : (Y;) 
ene” @ C Bo XY; Bi UX1 Y By LXy a (23) 


Sum of Square of Residue or Regression Independent (SQR) 


SQR = SQTOTAL - SQREG (24) 


7.3 DATA COLLECTION 
The quantitative research involves the collection of numerical data, 
the determination of the sampling having a classic separation of the types: 


The difference between qualitative-quantitative is of nature. 
While social scientists who work with statistics grasp only the 
"visible, ecological, morphological and concrete" region of the 
phenomena, the qualitative approach goes deep into the world of 
the meanings of human actions and relationships, a side that is 
not perceptible and cannot be captured in equations, averages 
and statistics (MINAYO, 2003, p. 22). 


Assuming that the data collection’, will provide a broader view of 
observed phenomena, aggregate a greater amount of information to be worked 
by statistical modeling. In this work we will use the secondary data divulged and 
the data codification of the international evaluations by OECD. 

According to some authors among them: Minayo (2003), Trivihos 
(1987), LUdke and André (1986), Engers (1994), Thiollent (1988), Fazenda 
(1994), Rey (1998), Gil (1987), affirm that the set of quantitative and qualitative 
data are not opposed, on the contrary they complement each other giving veracity 
to the research. Complementing Marconi and Lakatos (2003, p.190), they 
emphasize that "the technique of data collection to obtain information, uses the 
senses to obtain certain aspects of reality. 


4 The data will be collected according to the principle of triangulation described by Bazeley (2002), and 


also in Creswell (2003) and Greene et al (1989) adoption of mixed methods in data collection and 
results. 
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= Efeito de tratamento 
Analise da nao significativo 


As informacgées (Dados): jactaneia ee a 


Relatorios OCDE: 2000, 
2003, 2006, 2009, 2012, 2015, 


2018. 
Coeficiente de Variagao 





Efeito de tratamento 
significativo 


[ Regressao potinomial | ““"°° POSS°SLS | | Contrastes ortogonais 


on Qualitativo 








1. Mentalidade de crescimentos por | Sa 

caracteristicas dos alunos; Médias do PISA declarada 
2. Melhor desempenho entre Generos: Qualitativo nao pela OCDE em Relatorios 
desempeno em matematica e Ciéncia; Testes: Student 

3. Desempenho de leitura e gasto com estruturado Tukey e Sheffé 
Educagao; y 

Testes; Student, Tukey e Sheffé | 





Figure 22 - Flowchart of data treatments 
Source: Elaborated for this work 


The use of the statistical software Action Stat ** for regression analysis 
- in which we will verify if two or more variables are related in some way, to 
express this relationship establishing a mathematical model. This modeling is 
called Regression, helping to know how certain variables influence other 
variables, that is, it checks how the behavior of one variable(s) can change the 
behavior of another. ANOVA is a collection of statistical models in which the 
sample variance is divided into several components due to different factors 
(variables), which in the applications are associated to a process, product or 
service. Probabilities is the act of assigning weights to certain events. Thus, so 
that each one does not define probability of its form, we will assign a function 


weight that has some intuitive properties and finally Stochastic Processes. 
The National Center for Education Statistics - NCES, the official 


agency of the United States Government, maintains in DAS - Data Analysis 





43 The portal for researchers elaborated by Estatcamp, composed by qualified professionals, with 
masters 21d doctorates in statistics and computing. It currently has over 3 million views per year. 
Statistical Software with more than 80 thousand users. Among them: IMPA - Institute of Pure and Applied 
Mathematics, UFA - Federal University of Uberlandia and _ International Institutions. 
http://www.portalaction.com.br/ 


77 


System DATALAB a database of all information related to education and 


International Assessments. 


IES = NCES b3teition statistics = MeN 





DAS DATA ANALYSIS SYSTEM 
FROM DATA TO INFORMATION 


NOTICE: The DAS Online Application will be retired on 12/31/2020. 
Datasets available in the DAS will be made available in Datalab. 
WHATS NEW 


DAS Online Application to be Retired 
on 12/31/2020 


for vanous topics 





Tor exemple, 2 recent study found that 65.6 percent of 


Figure 23 - Data Analysis System 
Source: http://nces.edu.gov/das/ 


Having the following topics (1) Attendance and Enrollment; (2) 
History of Education; (3) Educational Transition; (4) Employment in Education; 
(5) Colleges and Functioning; (6) Education Finance; (7) Country and Family; (8) 
Pre-school and Fundamentalist Staff; (9) School and Institutional Characteristics; 
(10) Educational Law; (11) Special Education; (11) Personal; (12) Student 
Characteristics; (13) Teachers and Teaching. With regard to International 
Assessments, the data will be captured from IAS. 


IES * NCES Eitestion starictics 





IAP® international Activities Program (IAP) 
— > Join Newsflash wetances = 3 Contact Us 


bse Baie to) iota att Bye PLAAC Poa lela e Baeel elas Belg 


PISA results for financial literacy are in! 








talis 


ics + cits" 


Figure 24 - International Activities Program (IAS) 
Source: https://nces.ed.gov/surveys/international/ 


-2 
>> 
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7.3.1 Data organizations in documents 


As a way to systematize Data Collection to perform statistical and mathematical 
modeling of research work, we will use the International Student Assessment 
Program - PISA as an evaluation parameter among the nations researched in this 


thesis, as it is common in the USA, Portugal and Brazil. 


The figure 2144 below, the blue line corresponds to the average of the 
OECD member countries, in the black line shows the performance of Brazil in the 
evaluation, with the indication of the grade in each year (red) in each of the areas 
tested (Reading, Mathematics, Sciences). 


Reading Mathematics Science 
score points score points score points 
a 
4 15 
41 
403 414, 407 2 405 404 
400 40 400 
{ = @ 407 386 «389 384 402 401 
393 370° ° 390° 
377 
3 06° 50 
2003 2006 2009 2012 2015 2018 2003 2006 2009 2012 2015 2018 2006 2009 2012 2015 2018 
PISA PISA PISA 
@ Brazil OECD average —— Trend - Brazil 


Figure 25 - OECD chart evolution of PISA Brazil 
Source: OECD, PISA 2018 Data-base, Tables |. B1.10, I. B1.11 and I. B1.12. 


According to OECD* (PISA, p. 4) "average performance in mathematics improved 
in the period 2003-18 in the first cycles. After 2009, in mathematics, as in reading 
and science, the average performance tends to be stable.” 

Results of the International Evaluation Program in Portugal from 
2000 to 2018 (last two decades). 


44 Notes: indicates average performance estimates that are statistically significantly above or belowthe 
PISA 2018 estimates for Brazil.12 


The blue line indicates the average performance. Being the average of OECD countries with valid data in 
all PISA evaluations. The red dotted line indicates the average performance in Brazil. The black line 
represents a trend line for Brazil (best fit line). Source: OECD, PISA 2018 Database, Tables |. B1.10, I. 


B1.11 and |. B1.12. 
45 OECD (2019), PISA 2018 Results (Volume |): What Students Know and Can Do, PISA, OECD Publishing, 


Paris, https://doi.org/10.1787/5f07c754-en . 





79 


RESULTADOS DE PORTUGAL 


530 


510 ~ 


490 + 


470 ~ 





450 

—®— Matematica 

430 + —®- Leitura 

—®—Ciéncias 

410 “T T T T T T T 7 
2000 2003 2006 2009 2012 2015 2018 


Portugal obteve 492 ponto: Jf 


Pontuagao na escala PISA 








nos trés dominios, a par da média 


da OCDE 





Figure 26 - OECD PISA Portugal Average Evolution 
Source: IAVE - Institute for Educational Evaluation 


Portugal is one of the few countries that shows a trajectory of 
improvement in the last 7 (Seven) tests in all three domains above the OECD 
countries average. 
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Figure 27 - Evolution of PISA OECD Portugal tests 
Source: IAVE, OECD, PISA 2018 Base Date (2018, p.2) 


Between 2000 and 2018 Portugal is one of the countries that hada 
positive evolution of 22 (twenty-two) points in Portuguese, 17 (Seventeen) points 
in Science and 26 (twenty-six) points in Mathematics. 
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In the USA*®, the behavior in the evaluation of PISA in Reading, 
Mathematics and Science has always been bordering the OECD average. 


Readin : Mathematics Science 
score points 9 United States score points score points 
600 600 600 
550 550 550 
504 500 505 502 497 502 
500 ~ Seen gern —— 500 ~ 483-487-2843 500 - =< aA 
495 498 497 2 oe 496 
450 450 a8 470 450 
400 400 400 
350 350 350 
300 300 300 
2000 2003 2006 2009 2012 2015 2018 2003 2006 2009 2012 2015 2018 2006 2009 2012 2015 2018 
United States PISA United States PISA United States PISA 
@- United States ——- OECD average —— Trend - United States 


Figure 28 - Evolution of tests in the USA 
Source: OECD, PISA 2018 Database, Tables |. B1.10, I. B1.11 and I. B1.12. 


Figure 24 presents the average performance of PISA tests that 
integrate the universe of comparative evaluations of the educational system in the 
United States being part of INES*” - Indicators of Educational Systems. Only 
the performance in sciences in 2006 was significantly below the average score of 
2018, but even in sciences, the performance did not change significantly 
and followed a stable path since 2009. 





46 Indicates statistically average performance estimates that are significantly above or below the PISA 
2018 estimates for the United States. The blue line indicates the average performance of OECD countries 
with valid data in all PISA assessments. The dotted red line indicates the average performance in the United 
States. The black line represents a trend line for the United States (best fit line). PISA 2006 reading literacy 
results are not reported for the United States due to an error in printing the test books. 


47 The INES Project began in 1988 in partnership with the United Nations Educational, Scientific and 


Cultural Organization (UNESCO) and the World Bank, as a response to the need felt by politicians to have 
information that would allow them to compare the performance of their education systems with those of 
other countries and thus better evaluate and monitor the effectiveness and evolution of their education 
systems. Within INES a set of key indicators are developed, collected, analysed and interpreted for 
international comparisons, which are disseminated through annual publications: Education at a Glance and 
Education Policy Analysis. INES' work is developed through three networks: (a) network A, chaired by the 
United States, which develops result indicators on learning; (b) network B, chaired by Sweden, which 
develops social and economic result indicators on education; network C, chaired by the Netherlands, which 
develops indicators on school structures and processes. The European Union Statistics Office (EUROSTAT) 
and the qualitative analyses developed by the European Commission's information network for education 
(EURYDICE). From the INES meetings come recommendations of several orders for the study of PISA, 
comments are made and there is, since the creation of PISA, a strong link to the project of indicators. 
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8 STATISTICAL AND MATHEMATICAL MODELING 


Several statistical*® and mathematical*? models make up _ this 
modeling in the universe of sample research in OECD data. Several indicators 
are also used that report the size of the subpopulation and sample. The average 
value contemplated in this study is qualified in absolute and relative numbers. 
They are international evaluations made up of measures used in this thesis where 


the medians defined in this work are the midpoint of a characteristic. 


8.1 STANDARD ERROR CALCULATIONS ON MULTI-STAGE SAMPLES 

In PISA, as in IEA studies, the results are based on a sample. The 
published statistics are therefore estimates of means, standard deviations and 
regression coefficients. The uncertainty due to the sampling process should be 
quantified as standard errors, confidence intervals, P-value. 


Variation in student performance on the combined scale 


Countries Percentile Percentile | Percentile Gap |Percentile | Percentile| Percentile 
Sample Mean z 
mean Score 10th 25th 
5th 75th 90th 95th 
Brazil 210 36 5 6 6 oe 14 14 15 60 


Box 14 - Student Percentile Variation 
Source: Designed for this study based on OECD (2001). Knowledge and Skills for Life: First Results. 


Le l 

M=—> x, =— (6 +6+464...4144+144+15=10 =) 
N=" 36 

ho =+ Ace - 9? = Aa - 10? = 2 =5.833 (26) 
N ia 36 ia 36 


48 There are many tested and proven methods for educational data analysis; techniques commonly 
found in other sectors are used to improve the state of analysis in education, in particular, going beyond 


exploratory data analysis, introducing more advanced techniques such as inferential statistics and 
predictive modeling (WICKHAM, HENRY & MULLER, 2020). 

49 Mathematical modeling seeks to model a certain phenomenon of reality in order to understand this 
phenomenon. It can also be defined as: free and spontaneous, it arises from man's need to understand 
the phenomena that surround him in order to interfere or not in his process of construction. 
https://www.somatematica.com.br/artigos/a8/p2.php 








Modeling for Calculation - Standard Deviation for Multiple Stage Samples 
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Box 15 - Average Population Estimate for all possible Samples. 


Source: Designed for this study based on OECD (2001). Knowledge and Skills for Life: First Results. 


Number of possible sample size n of a population size N 


The distribution of the sample variation in equation 25 has: a mean of 10 


[(2*5.5)+(4*6)+.....(4*14) +(2*14.5)| 
ny 10 


a Standard Deviation (SD) of 1.7 


* 630 





The DP of a sample distribution is denoted Standard Error (SP) 


cy = TEE ao 


The sampling distribution in the mean seems a normal distribution: 


83 


(27) 


(28) 


(29) 


(30) 








Figure 29 - Standard deviation. 
Fonte: Adaptado para este estudo. OECD (2001). Knowledge and Skills for Life: First Results. 





Presenting the number of samples with an average included 


between: 

[(10-1.96 EP); (10 + 1.96 EP)] 

[(10-3,30); (10 + 3,30)] 

[6,70; 13,30] 

There are: 16°° + 28 + 38 + 52 + 60 + 70+ 60+ 52 +38 + 28+ 16 = 598 


samples, therefore 94.9% of all possible samples. 


With a population N (10, 5, 83), 95% of all possible size 2 samples will have an 


average estimate of the population included between 6.70 and 13.30. 


50 Parameters established in Table 14 as Sample Average 7.0 going up to average 13.0. 


84 


8.2 SAMPLE DISTRIBUTION OF MEAN ESTIMATES 
In this context of modeling study, we have the sample distribution of 
the mean estimates of all possible samples of size 4, enlarging the spectrum of 


the subpopulation researched in the international evaluation PISA. 
5000 
4500 
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1000 
500 

















Figure 30 - Distribution of Possible Samples size 4. 

Source: Designed for this study based on: OECD (2001). Knowledge and Skills for Life: First Results. 
The distribution of the variation of the sample an average of 10, from equation 
(25). 


Lh _[G*6) +10*6.25) +......0*13.75) +3*14)] ae 
| 58905 (31) 





Considering a standard deviation of 1.7 





, \o- 10) -H6- 10)*+(6- 10) +.......04- 10) 

















Oy = =1.335 
58905 (32) 
Fr 
() =1 155 (33) 
Sample Variance Mean Inversely proportional to Sample 
Size 
02 Students - Score 20 5,5 to 14,5 
04 Students 6.0 to 14.0 
10 Students 7.0 to 13.0 
02 students 11.0 to 
- Score 40 29.0 














Box 16 - Variance of the mean sample is inversely proportional to the sample size. 
Source: Adapted for this work: (OECD 2005) PISA 2003 Technical Report. 
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8.3 RESEARCH AND CALCULATION OF SAMPLING VARIANCE 


For analysis of the weights used in the PISA 2001 and 2003 exam to 


model the data and make inferences and sample error. Variables are calculated 


as: evaluated, ineligible and excluded students, performing standard error 


estimates, significance test and delineating confidence intervals, in the PISA 


sample design in each individual country. 


According to Cochran (1977); Sarndal et al (1992), the procedures 


used to derive the research weights for PISA reflect best practice standards for 


analyzing complex research data®' and the procedures used by the world’s leading 


statistical agencies. 





Level 


Lower 
Limit 


Reading level requirement 





625,6 


At Level 5, students are able to complete sophisticated reading items, 
such as those related to the use of hard-to-find information in texts with 
those they are not familiar with; show a detailed understanding of these 
texts and infer what information in the text is relevant to the item; critically 
evaluate and establish hypotheses; draw on expert knowledge; and 
include concepts that may be contrary to expectations. 





552,9 


At Level 4, students are able to respond to difficult reading items such 
as situate aggregate information, interpret meanings from subtleties of 
language, and critically evaluate a text. 





480,2 


At Level 3, students are able to manipulate reading items of moderate 
complexity, such as placing multiple fragments of information, linking 
distinct parts of a text, and relating it to familiar everyday knowledge. 





407,5 


At Level 2, students are able to answer basic reading items such as 
locating direct information, making easy inferences of various kinds, 
determining what a well-defined part of a text means, and employing a 
certain level of external knowledge to understand it. 











334,8 





At Level 1, students are only able to respond to the less complex reading 
items developed for PISA, such as locating a piece of information, 
identifying the main theme of a text, or establishing a simple connection 
with everyday knowledge. 








Box 17 - OECD Proficiency Levels reading 
Source: Anexo A - PISA 2003 Technical Report describes those methodologies 





51 To conduct a Complex Survey, there is a need for established standards (proficiency levels) and indexes. 
With the use of probability sampling proportional to size, in countries with few selections of schools for 
sure and without over or under-sampling of any explicit stratum, the weighted and unweighted rates are 
very similar (ANNEX A p. 169). 
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Level 


Lower 
Limit 


Requirement of mathematical level 





669,3 


At Level 6, students are able to conceptualize, generalize and use 
information based on their investigations and the modeling of complex 
problems®2. They can relate different sources of information and 
representation and translate them among themselves in a flexible way. 
They are able to demonstrate advanced mathematical thinking and 
reasoning. In addition, they can apply this understanding and knowledge 
along with the dexterity for formal and symbolic mathematical operations 
to develop new approaches and strategies to face new situations. They 
can accurately formulate and communicate their actions and reflections 
RESPECT discoveries, interpretations and argumentations, and adapt 
them to new situations. 





607,0 


At Level 5, students can develop and work with complex situation 
models; identify boundaries and specify assumptions. They can select, 
compare and evaluate appropriate problem-solving strategies to address 
complex problems related to these models. They can work strategically 
by making extensive use of well-developed thinking and reasoning skills; 
representations by association; symbolic and formal characterizations; 
and the understanding of these situations. They can formulate and 
communicate their interpretations and reasoning. 





544,7 


At Level 4, students are able to work effectively with explicit models for 
complex concrete situations that may imply limitations or require 
assumptions to be made. They can select and integrate different 
representations, including symbols or associate them directly with real 
world situations. They can use well-developed skills and reason with 
some understanding in these contexts. They can construct and 
communicate explanations and arguments based on their interpretations 
and actions. 





482,4 


At Level 3, students are able to perform clearly described procedures, 
including those that require sequential decisions. They can select and 
apply simple problem solving strategies. Students at this level can 
interpret and use representations based on different sources of 
information, as well as reason directly from them. They can generate brief 
communications reporting their interpretations, results and reasoning. 





482,4 


At Level 2, students can interpret and recognize situations in contexts 
that require only direct inferences. They can extract relevant information 
from a single source and make use of only one type of representation. 
They can employ algorithms, formulas, conventions or basic procedures. 
They are able to reason directly and make literal interpretations of results. 











357,8 





At Level 1, students are only able to answer questions that present 
familiar contexts in which all relevant information is present and the 
questions are clearly defined. They are able to identify information and 
develop routine procedures according to direct instructions in explicit 
situations. They can perform actions that are obvious and follow them 
immediately from a given stimulus. 





Box 18 - OECD Mathematics Proficiency Levels. 
Source: Anexo A - PISA 2003 Technical Report describes those methodologies. 


52 A mathematical model, in turn, consists of a conceptual, descriptive or explanatory system, expressed 


by means of a language or a mathematical structure, with the purpose of describing the behavior of 
another system and allowing the realization of predictions about this other (LESH; CARMONA; 
HJALMARSON, 2006, apud ALMEIDA; VERTUAN, 2011, p. 21). 








Level | Lower 
Limit 





Demand of the Science level 
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At Level 6, students can safely identify, explain, and apply scientific 
knowledge and insights* into a wide variety of complex life situations. 
They are able to relate different sources of information and use evidence 
drawn from such sources to justify decisions. They demonstrate clearly 
and consistently an advanced scientific thinking capacity, and 
demonstrate a willingness to use their scientific knowledge to solve new 
scientific and technological issues. Students at this level can also use 
scientific knowledge and develop arguments’ to _— support 
recommendations and decisions focused on personal, social and global 
situations. 





At Level 5, students are able to identify scientific components in many 
complex life situations, to apply both scientific concepts and science 
knowledge to these situations, and are able to compare, select and 
evaluate appropriate scientific evidence to respond to life situations. 
Students at this level can utilize well-developed research skills, relate 
knowledge appropriately, and reflect critically on situations. They are also 
able to construct explanations based on evidence and arguments based 
on their critical analysis. 





At Level 4, students are able to work effectively with situations and 
issues involving explicit phenomena that require them to be able to make 
inferences about the role of science and technology. They are able to 

select and integrate explanations from different disciplines of Science or 
Technology and relate them directly to aspects of life situations. They can 
reflect on their actions and communicate decisions using knowledge and 





| scientific evidence. 
At Level 3, students are able to identify clearly defined scientific issues 


in a variety of contexts. They can select facts and knowledge to explain 
phenomena and apply simple models and research strategies. They can 
interpret and use scientific concepts from different disciplines and apply 
them directly. They can also discuss facts and make decisions based on 
scientific knowledge. 





At Level 2, students have reasonable scientific knowledge to provide 
scientific explanations in family contexts or to draw conclusions based on 
simple research. They are able to reflect directly and make literal 
interpretations of scientific research results or technological problem 
solutions. 








6 707,9 
5 633,3 
4 

558,7 
3 484,1 
2 409,5 
1 334,9 








At Level 1, students have limited scientific knowledge, so that they can 
only apply it in a few family situations. They are able to present obvious 
scientific explanations and draw conclusions from explicitly presented 
evidence. 








Box 19 - OECD Science Profiency Levels. 
Source: Anexo A - PISA 2003 Technical Report describes those methodologies. 


According to Tavares (2009) three models of the Teaching System, exposed by PISA 
ina rank: Model -1 Brazilian Teaching System n+1; Model - 2 Portuguese Teaching System n-1; 


Model - 3 United States Teaching System n=0. Presenting the result as Binomial Theorem 


by this principle. 


53 It is worth pointing out here that an individual considered scientifically literate is one who not only knows how 


to read scientific vocabulary, but understands the scientific processes related to his or her everyday life, the social 


problems related to science and technology and their impact on society, in order to be able to participate in 


decision-making processes on issues involving science. 


88 


Coverage of the national desired population, calculated by P / (P + E) x 3 [c] / 3[a]. 
In Science, Mathematics and Literacy. 

The desired national population (PDN), defined in table 16 has an 
answer [ 5 levels ] and denoted here as Literacy ("desired target population" ), 
is the population that includes all 15 year olds enrolled in each country in the 7th 
grade and above (withthe possibility of small levels of exclusions),based on national 
statistics. However, the final NDP reflected in the sampling structure of schools in 
each country may have had some school-level exclusions. The figure representing 
the population of 15-year-old students enrolled minus those in excluded schools 
is nitially represented by the answer box [c] It is denoted here as 5 [levels from 625.6 
to 334.8] (and in which very small schools have only one or two eligible students 
could not be excluded from the school board, but could be excluded in the field if 
they still had exactly one or two eligible students at the time of data collection, so 
what is noted in level 1 as level 5[c] is anumber that excludes schools excluded 
from the sampling frame, in addition to schools excluded in the field, so the term 3 
[c] /3 [a] provides the proportion of the PDN covered in each country based primarily 
on national statistics . 


The value (P + E) provides the weighted estimate from the student sample of 
all eligible 15-year-olds in each country, where P is the weighted estimate of eligible 
non-excluded 15-year-olds and E is the weighted estimate of eligible 15-year-olds 
that were excluded within schools. Therefore, the term P/ (P+E) provides an 
estimate based on the student sample of the proportion of the eligible 15-year-old 
population represented by the non-excluded eligible 15-year-olds. 


In mathematics, table 17 is defined by 6 levels with value (P + E) providing 
the weighted estimate of the sample of students of all eligible 15 year old students in 
each country, where P is the weighted estimate of the eligible 15 year old students not 
excluded and E is the weighted estimate of eligible 15 year olds who have been 
excluded from schools. Therefore, the term P / (P + E) provides an estimate based on 
the sample of students of the proportion of the eligible 15-year-old population that is 
represented by eligible 15-year-olds not excluded. 


The national enrolled population (NEP), defined in Table 18 related to Science 
[b] and denoted here as 6 [levels ]( and as "enrolled at 15 years of age"), is the 
population that includes all 15 enrolled children of 1 year in each country in 7th grade 
and above, based on national statistics. defined in models 1, 2 and 3 (n + 1 Brazil), 
(n-1 Portugal ) and (n=0 United States ) final, denoted here as 3 [c] as described 
above for the coverage index from 1 to6, reflects the 15-year-old population after 
school-level exclusions. This figure represents the population of enrolled young 
people aged 15 minus those from excluded schools. not excluded. 
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n n 
Theorem™: For all the value of n natural, one has: (4 +.x)"= Yk=0 (ak (34) 


Whatever the actual value of x 
Demonstrating Model 3 - American System n =0 


0 0 
0 
Lk=o Cs = (0),02 1 e(1+x)°=1, logo (35) 
0 
0 
(1+x)° = Li=0 (i) (36) 
We now admit that the theorem is valid for n, i.e. that 
n n 
(14+x)" = De=0 Cx (37) 
We will demonstrate what equality is like for n+1 
(1+x)"*1 = (14x) (1+x)"comes (38) 
7 n n n n 
(14x91 = (44%) Bho (jy = U0 (yak + (ie)ynet (39 ) 


nt+1 


me td 
Manipulating the second member (right side) until you get > ( k ) in fact, 


("a ltl Gentle War 


But, as 


(® ; leak : ary Co lee ae i s (41) 


n 
n 
We proved as we intended, that (1 + x)"= Xk=0 (ie) xk and so we finish the 


demonstration of the American Educational System. 
From the development of (1 + x)" it immediately follows that (x+y)". 


Corollary: Whatever the real x and y and the natural n is valid the formula: 
n 


(x+y)? = yK=0 (.) xkyn-k (42) 





*4In mathematics, a theorem is a statement that canbe proved to be true, through other 
statements already demonstrated, like other theorems, together with previously accepted 
statements, such as axioms.http://www-di.inf.puc-rio.br/~poggi/discrete.pdf 
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Figure 31 - Total cost of Education in the USA. 
Source: https://gradepowerlearning.com/cost-of-education-in-usa/ . 
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Thus, Model 3 - US Educational System represented n=0 and by 
mathematical modeling (x+y)". It is equivalent in schooling time; Pupil- Teacher 
Ratio; System Performance (approval, disapproval); Investment in Public 


Education (GDP); Enrollment and Teachers. 
Demonstrating the Model - 2 Education Systems of Portugal n-1. 
We admit that y # 0: 


As, for y = 0 you have (x + y )"= xn and 


TL n 
k=0 a Xkyn-k _ x" = (x + y)o (43) 


The principle of mathematical induction described by Tavares (2009) 
and the following axiom of Peano °° says that: if the set A, contained in N, is such 
that 1 belongs to A and n-1 whenever n is an element of A, then A= N[N _ here 
is the set of naturals 1,2,3....]. 

A P property is said to be hereditary®® when, being true for the whole 
n, it is also true for the successor of n (n+1). 

Thus, the principle of induction is equivalent to affirming that a date 
proposition, true for n=1 and hereditary, implies that it is true for all naturals 
V2 det 

Therefore, the application of this method composed of two steps or 
known steps are: 

1. Demonstration that a given proposition is valid for n=1 (base case); 

2. Demonstration that the proposition is hereditary (Induction Stage). This 
principle has nothing or almost nothing to do with the method of induction proper 
to the natural sciences, which is characterized by establishing a general law 


observing the repetition of the same phenomenon and numerous cases. 





55 Giuseppe Peano, is an Italian author who in 1888 introduced the axiomatic definition of vector space, 
calling it linear systems. Peano's axioms were first formulated in 1889 in the Arithmetices Principia nova 
methodo exposed, which represented an attempt to reduce common arithmetic to pure formal 
symbolism. Peano expressed the postulates in symbols, instead of the words we use. The postulational 
method reached a new level of precision, without ambiguity of meaning and without hidden hypotheses. 
He also developed symbolic logic. 

6 In 1890, Peano showed that mathematics could surprise common sense when he built continuous curves 
that fill space - that is, curves given by parametric equations x = f(t), y = g(t), where f and g are real 
continuous functions in the interval 0 $< t <1, whose points completely fill the unit square OS x<1,0Sy 
<1. This paradox matches perfectly with Cantor's discovery that there are no more points in the unit 
square than in the unit line segment. However, in 1903, Peano became distracted by the invention of the 
international language he called Interlingua or Latino sine flexione, with vocabulary taken from Latin, 
French, English and German. This movement was more ephemeral than his axiomatic structure of 
arithmetic. http://www.matematica.br/historia/peano.html 
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Still according to Tavares (2009), not all induction tests®” have the 
same degree of difficulty. While modeling (43) is extremely simple and natural for 
the U.S. Educational System; in model - 2 which presents the Portuga 
Educational System, modeling an auxiliary algebraic identity to be used in the 
induction step, it can be deduced from Ruffini's®® rule of division of polynomial into 


x Investment, performance n, by x - a. 


Example 1: Investment in Public Education - Mathematical Induction Test 


_—- * (44) 
k=15k = 1-97 
x : 
For n =1 the equality is verified: 2 = 1 2 (45) 
It is admitted that it checks for n 
we «=f 1 
ae (46) 


It is proven that in the verified investment time for n+1, that is, we 





: n+11 1 
should arrive at’) ee (47) 
So, adding anti To both factors®®? we have the members of equality and 


simplifying the second member, we deduce successively. 


ml 


1 Hl ; 1 1 i 1 , 1 
= + gnt1 =1~oa ? oan + (=-sex) =1-a 
k=1 





57 Mathematical induction is a method of mathematical proof used to demonstrate the truth of an infinite 
number of propositions. The simplest and most common form of mathematical induction proves that a 
statement is valid for all natural numbers n and consists of two steps: 1. The base: show that the statement 
is valid for n = 0, or n = 1, depending on the definition used of N; 2. The second inductive step: show that 
if the statement is valid for n =k, then the same statement is valid for n = k + 1. Available: 
http://www.obmep.org.br/docs/apostila4.pdf Also in Tavares, 2009. 


58 Briot-Ruffini's algorithm, sometimes called Ruffini's rule only, is a method of solving polynomial 
fractions, created by Paolo Ruffini. This algorithm consists in making the division by calculating only 
coefficients and only serves for divisions of a polynomial by a binomial. The divisions of polynomials by 
binomials, for example: (x-2), (x+3/2) and (x+5), appear in mathematical problems more frequently than 
any other polynomial divisions and play an important role in the search for function zeros and equation 
solving. Available: http://www.rom.org.br/cdrom/34/3.htm 


5° Factors studied in the Education System by this thesis are: (1) Time of Schooling; (2) Teacher-Student 
Ratio; (3) System Performance; (4) Investment in Public Education; (5) Enrollment; (6) Teachers. 
https://meuartigo.brasilescola.uol.com.br/direito/o-sistema-educacional-brasileiro.Atm 
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Thus, we have the application of Models 1,2 and 3 in Educational Systems: 





gn+i on one+k 


(> = Escolarizacao i Professores Desempenho 4 — —) i Investimentos 


Qn gnet gn+l 


Now, we prove in this way that equality is for any whole n = 1. 


Example 2 - Performance of the Educational System® if n is whole positive, prove 
Forn= 1, a” = b” = (a — b) sh ak pert (48) 
at —b1 = (a—b)Yithak bt tk =a—b (49) 


Before applying the induction hypothesis, for Tavares (2009) the fundamental 
idea is to show the validity of the auxiliary identity 


(a” _ b”) P(n + 1) — tat tt —_ b™*1)P(n) (50) 
Where sg 
P(n) = - aie + (51) 
k=0 
Suit 7 
(a” +h") P(n+ 1) — (a” = by) akpr-k (52) 
k=0 
n=1 _ ,n+1 _ (a-b)P(n) 
a b = a P(n+1) (53) 
= (a— b)P(n+1) (54) 
n 
= (a = b) >, ak peek (55) 
k=0 





60 RAP - Student Teacher Ratio. Number of average teachers per student is a significant problem. 

As performance of the Education System we have: the quality of Education measured by Evaluations 
such as: ANA - National Literacy Evaluation; IDEB - Basic Education Development Index; SAEB - Basic 
Education Evaluation System; Prova Brasil; Enem; All described in Chart 5 of this thesis, including PISA - 
International Student Evaluation Program. 
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Example 3 - Time of Schooling®' proving that there is only one natural number 


that is verified in the relation Mp = 2n2 — 4 (56) 


209 3.3 9 = 0 ,22016 - 2018 __ | (57) 


Also in Tavares (2009) we found using the principle of induction to 


prove that the relationship is not satisfied by any natural 


Example 2 - Demonstrating the Range of Performance of the Educational System 
4) BSS. n—X) 58 
ier) =e we 


n per 1 (quality measured in Test or Evaluation) 


Which we consider to be true, since from the functional equation of gamma 


function: Px +1) =2F() 159) 


1 
You get, for x =5 


1 a A 1 

ae (60) 
r(1+ 5) a) A ue 
1 
Transforming the functional equation, x =n ty As it comes successively (61) 
1.3.5...(2n—1)[2(n +1) -1] 

- eee (62) 
PISA.PIRLS.TIMSS.PIAAC.T ALIS (2n—1)[2(n+1)- (63) 

= gnt+1 


The performance of the education system described in equation (63) framed as 
quality points to the results obtained in tests and evaluations. 


61 Schooling time. The average number of years of study in Brazil increased between 2016 and 2018. The 
indicator went from 8.9 years to 9.3 years in 2018. Since 2016, this average has grown by 0.2 years per 
year. Among women it was 9.5 years, while among men it is 9 years. 


95 


9 PRACTICAL DEFINITIONS AND CONSIDERATIONS AS MODEL 
In this Chapter we will be presenting the results of metrics measured 
in International evaluations compared to actions carried out in the Educational 


Systems of countries that obtained favorable results in these tests. 


9.1 MODELING THE ANALYSIS OF VARIANCE 
We present the model proposed for ANOVA using the values 
calculated by PISA in the countries shown. 





“Score Anos do PISA - Brazil Overall| Content 


2000 | 2003 2006 2009 | 2012 | 2015 | 2018 
01 404 











4oo] - | - | 390 | 405 ca 4 Ea Science 


| 
400| 396 | 403 | 393 | 412 413 | 2831 | Literacy 
fee] 


407 | 407 | 413 
350| 356 | 370 | 386 | 389 | 377 8 2262 is 


Box 20 - Values calculated in the PISA evaluation by Brazil 
Source: PISA/OECD prepared for this work according to figure 21 





C = 7.0952/21 = 2.397.096,43 
SQA = (1/24) x (20022 + 28312 +2622) - 2.397,096.43 = 10,901,108.74 
SQ plots 2000) = (1/2) x (8962 +3562) = 205,144 


SQ (plot 2018) = (1/3) x (4042 + 4132 + 3842) = 372,430 
SQR\a) = C - SQ plots 2000 to 2018) 2,397.096 - 2,133.795 = 263.301 
SQTOTAL(w = 263,301 














Causas Grau Soma dos Quadrado_ F calc F trab 
de variagdo de Liberdade Quadrados Meédio [5%] 
Fator A | 3 |__372,43 | 263,301 | 6,45* | 4,07 
Residuo | 8 | 205,144 | 167,29 | 1,62 | 2,59 
Fator B | 10 | 2.397.096,43 | 430,59 | 7,75* | 3,62 





*Significant at 5% error probability level; 
Box 21 - General Variance Analysis Chart 
Source: Developed for this research 


The analysis of variance shows that for factor A and B, test F was 
significant at a 5% error probability level, concluding that there is a difference 
between the years analyzed (2000 to 2018), there is still a contrast between the 
means of the levels of factors A and B statistically different from zero at the 5% 
probability level. 


The results described in table 20 reveal that the 6 (six) PISA tests in 
the years [2000; 2003; 2009; 2012; 2015 and 2018] had a statistically significant 
correlation, but in general, there was no statistically significant change in the 
quality of education in Brazil; it is essential to note that correlation is not equal 
to causality. Since the resulting model had moderate statistically significant 
strength based on r-square values, it is plausible that the research needed to 
analyze each variable independently for a deeper dive policy analysis that can 
influence the complexity of inequalities in education. 

In the United States Shavit, Arum, Gamoran, & Menachem,( 2007), 
present a study on teaching stratification. We also have in McFatter (2017), the 
demonstration that the ANOVA model allows a ®*complete analysis of comparative 
educational research among the variables. As a model in this work the emphasis 
will be on the Educational System. In the case of Brazil, 6 (six) variables are 
highlighted in table 21. 
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Schooling Performance Enrollment Investments Teachers Reason Student-teacher 





_ Frequency Quality Quantitative __E. Infant Salary Quant. Prof. per student 
_ Dropout _ Assessments Target _E. Elementary Training Inefficiency 
| test Studies medium Workload Efficiency 
_Absenteeism E. Technician 





‘Studies Superior 





Postgraduate o 





% PIB 





Box 22 - Variables of the Educational System studied 
Source: Prepared for this work 


62 For qualitativists, the reality is constructed by the individuals involved in the research, it is subjective, 
multiple and only understood by examining the perceptions of the investigated subjects and those who 
interpret the phenomenon. For quantitative researchers, the researcher must be independent and must 
also keep his distance from what is being investigated, striving for objectivity, avoiding bias, ensuring the 
representativeness of the analysis through the selection of the sample (CRESWELL, 1994). 
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9. 2 ANALYSIS OF THE EDUCATIONAL SYSTEM 
According to Jewell (2010) education systems are influenced by 
communication and exchange of information, ideas, funding, research and so on 


by academics, professionals and governments. 


PRE-SCHOOL FUNDAMENTAL MIDOLE HIGHER 
EDUCATION EDUCATION EDUCATION EDUCATION 
HIGHER 


BASIC EDUCATION EDUCATION 


VOCATION AND TECHNOLOGICAL EDUCATION 


Initial and Continued 
Education 


{EJA) : ; 
Technical Education 


Higher Education 
(Technologist) 


Figure 32 - Conceptual Structure of the Educational System 
Source: http://portal.conif.org.br/en/international-conif-en/educational-system 


We also find in the structure of the educational system® two 
conceptual flows. 


The structure of the Brazilian educational system is defined by 
two main legislatures. They are the Education Guidelines and 
Bases Law - Law No. 9,394 of 1996, known as LDB - and the 
general guidelines of the 1988 Federal Constitution - which within 
Chapter Ill determines that basic education is a right of all 
citizens. These guidelines authorize the government spheres to 
conduct and maintain educational programs, which are thought 
from the Common National Curricular Base (BNCC) (CAMPIONI, 

2018, p. 2). 


As a description of the educational system, besides the structures 
described in LDB, we also have several agencies at the federal level: MEC; CNE. 
At the State level: SEE (State Education Secretariat); CEE (State Education 
Council). At the State level: SME (Secretaria Estadual de Educagao); CME 


(Conselho Municipal de Educagao). 





63 Main feature of the Portuguese Educational System described in Eurydice/ OECD/European 
Commission. Dttps://eacea.ec.europa.eu/national-policies/eurydice/content/portugal_en 
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In the figure below we describe the characteristics of the 3 (three) 
Educational System, Brazil, United States and Portugal. 
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Figure 33 - Comparison of Educational Systems 
Source: https://eacea.ec.europa.eu/national-policies/eurydice/national-description en (adapted) 





The educational systems described in figure 29 are divided into: 
years, grades, ages and stages of schooling. For comparative analysis we will 
use the description in chart 21 with the 6 (six) structured variables to be studied. 


Apparently the three educational systems are the same, only the 
secondary variables differ from one country to another, being: Enrollment rate, 
high graduation rate, student financing, teacher quality, high literacy rate among 
youth and adults, and gender variations (0 = male, 1 = female, 3 = immigrant, 4 
= black and 5 = white). 


For Gelman (2008) the correlational research employed here measured 
the degree to which two or more variables are related. In this method the 
theoretical reference quali-quanti are analyses of data sets produced by 
OECD/PISA. 
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According to Klein (2011), there is a problem of comparability of 
results in PISA, using the TRI® - Item Response Theory, with the Rasch® 
parameter, are presented as divergences: (1) Beginning of the school year; (2) 
School age ratio; (3) Different school age grades; and (4) Age from 15 years and 
03 three months to 15 years and 02 two months - predicted in the OECD has no 
relation with school age according to the Standardized Classification of Education 


(CINE) &€. 


























Country Quantitative Qualitative 
Studied Scale? Average | Interdecil* Level Requirement Indicators 
Brazil (L) 55-59 413 286-548 ||21,5 Basic e 18% na Average(2) Regular 
EUA (L) 10 20 505 361-643 75% Basic e 15% na Average(5)| Excellent 
Portugal (L)| 20-26 492 362-613 32,1% Basic e 65,7% Average(4) slapping 
Brazil (M) | 69-72 | 384 277-501 || 31,8% com nivel basic (2) Regular 
EUA (M) 32-39 478 357-598 | |25,5% Basic e 45,3% Average(4) | slapping 
Portugal (M)| 23-31 492 362-614 | |55,8% basic e 38,7% Average(4) | slapping 
Brazil (C ) 64-67 404 292-527 ||45% com pior desempenho (1) | Insufficient 
| EUA(c) | 12 23 | 502 [| 371-629 ||63,2% Basic e 32,4 Average(5) | Excellent 
Portugal (C)| 21-29 | 492 362-613 | |36,7% Basic e 47,9% Average(4) | slapping 














2.PISA Score Scale 4.Interdecil is a measure of the statistical dispersion of values in a data set 


Box 23 - Comparison between Quali-quonti 
Source: OECD/PISA 201 8(Adapted for this Work) 


The comparative table presenting the indicators of Brazil on PISA 
2018 has the following notes: in Letramento (Regular); in Mathematics (Regular) 
and Sciences (Insufficient). We also find the USA and Portugal mentioned among 


the countries studied. 





°4 Ttem Response Theory (TRI) is a way to analyze the answers to tests or questionnaires with the objective 
of improving the accuracy and reliability of the measurement. If you want your test to actually measure 
what it should measure (i.e., mathematical ability, a student's reading ability, or historical knowledge), TRI 
is a way to develop your tests. https://www.statisticsho wto.com/item-response-theory/ 

®5The Rasch model, named after Georg Rasch elaborate a psychometric model designed to analyze 
categorical data, such as answers to questions in a reading evaluation or answers to questionnaires, the 
Rasch measurement is implemented with Rasch maximum probability estimation together, partial credit 
and rating scale model parameters by means of a proportional curve fitting algorithm. 

66 The CINE (whose acronym in English is ISCED - International Standard Classification of Education) is an 
instrument produced by the United Nations Educational, Scientific and Cultural Organization (UNESCO) to 
serve as a reference in the compilation and analysis of educational statistics, assisting countries in the 
systematization of data at different educational levels and in international studies (UNESCO, p. 5, 2015). 
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9.3 PROPOSALS FOR BUILDING THE EDUCATIONAL SYSTEM 


The Educational Systems analyzed for construction in this thesis, are 


considered as dynamics, As an example Brazil vs USA, presented below. 


4 year college course senior 
Ensino Superior Bachelor's junior 


Graduaybo sophomore 
freshman 
12th 

Tith 

10th 
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7th 
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4th 

3rd 

2nd 


Educac&o Basica 





Série tdade/Age Grade 


Figure 34 - Brazil vs USA Educational System 
Source: NCES/ https://www.studyusa.com/pt/a/28/o-sistema-de-educacao-americano 








The differential between the two Systems, besides the fact that K12 
is compartmentalized (equivalent to basic and middle education in Brazil) are the 
specific Public Educational Policies, among them: (NCLB) the No Child Left 
Behind Law (2002); ESSA®°®- Every Student Succeeds Acts (Figure 17); 
Comparative Education; Alternative Certification to be a_ teacher; 
Multicultural Education (Whites, Blacks and Immigrants); Continuing 
Learning among others... 

In the educational model of Portugal presented in the figure below, 
the same 12 years of schooling (Basic and Secondary Education) of the Brazilian 


66 All Successful Students Act (ESSA) throughout the USA. https://www.ed.gov/essa?src=rn 
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and American System (K12) are verified. However, they are divided into Cycles 


and Courses (secondary). 








Figure 35 - Complete Portuguese Educational System 
Source: GIASE (2005) School groupings: national summary table. 


Some factors were found that influenced the progressive increase of 


evaluation in education in 


Portugal with PISA. Considered by the consistency of 


the results as the "Rising Star of International Education" informally, some 
authors indicate elements in the basic structural reform that provided this 


evolution. 


Fernandes (2007) highlights characteristics of the evaluation system 
in Portugal that directly influences PISA. 


Summative assessment, designed to classify and certify 
students, should occur only to make global assessments of what 
students know and are able to do. 

Formative evaluation should be an integral part of teaching and 
learning and be related to: a) self-evaluation and self-regulation 
of learning by students; b) the use of a variety of evaluation 
strategies and instruments; c) the participation of students and 
various stakeholders in the evaluation process; d) the 
transparency of procedures; e) the definition of criteria relating to 
skills to be developed; and f) the feedback that teachers should 
provide students in a_ systematic way after evaluation 
(FERNANDES, 2007, p. 588). 


This evaluation system described has impacted the progressive increase of 


PISA. 


According to the authors Escola, Catarino & Aires (2018) the 


influence of the Escola Nova Movement in the teaching of the mother tongue, 
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had as its main objective to develop in the students the ability to express 
themselves orally with correction and dexterity in writing, in order to know how to 
write documents such as letters, telegrams, receipts, declarations, requirements 


and expositions, so necessary to their daily lives. 


As far as composition is concerned, an integral part of the Mother 
Tongue program from Grade 2 onwards, the authors’ concern in 
preparing students for this task in Grade 1 is already well known, 
specifically in the acquisition of vocabulary that they consider, at 
this stage, to be very simple and limited. Still the simple writing 
exercises should continue in 3rd grade and it is indicated that 
they should consist of "summaries of the lessons when they are 
appropriate", and students are required to "judiciously use 
punctuation and accentuation". It is during the 4th grade that the 
written compositions acquire a more important status, and should 
be "intensified and graduated in difficulties", starting with themes 
suggested by the teacher and then dealing with free subject" 
(SCHOOL, CATARINO & AIRES 2018, p. 2016). 


The practice described by the authors has influenced and influences 
even today, directly on the increase of the literacy of students in the first cycle. 


Pintassilgo and Oliveira (2013) present the Teacher Training Model 
at the undergraduate level®’ as acontinuation of the Bologna Process 
68 described as solid and consistent in content and effective when it comes to 
compulsory internship. Containing two Stages, Decree-Law no. 43/2007 explicitly 
assumed the purpose of defining the conditions necessary to obtain professional 
qualification for teaching in a given field (Training) and the 2nd Cycle of Bologna 
in the Portuguese case, to make the designation of a Master’s degree, as 
according to the legislator, raise the level of qualification of the teaching staff and 
enhance the "respective professional partner status". 

Still according to John Nordahl of the OECD (2018 p. 2) "The quality 
of a country’s education will never be greater than the quality of its teachers, 


67 The degree (1st cycle) has a normal duration of six to eight semesters and grants between 180-240 ECTS 
credits. This level of studies provides students with basic knowledge in the scientific areas of the course 
and instrumental and systemic skills important for their immediate employability or for the pursuit of 
higher level studies, namely master's degree (2nd cycle). 


68 The Bologna Process was named after the so-called Bologna Declaration, which was signed on June 19, 
1999, in the city of Bologna (Italy), by the ministers responsible for higher education from 29 European 
countries, including Portugal. It is a process of intergovernmental reform at the European level that is part 
of the objectives of the Lisbon Strategy and aims to achieve the European Higher Education Area. 
https://www.uc.pt/candidatos-internacionais/sistema_graus/processo-bolonha 
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in Portugal the teacher training model is considered hard line, with a rigorous 
admission test for the best students to enter teaching. The training for a basic 
education teacher lasts from 6 to 7 years and to take over the classroom needs 
to go through a National certification process that attests that the requirements 


have been met. 


An educational policy conducted by Portugal that impacted on the 
country’s education was in the 1970s the universalization of education to all 
children of school age, requiring schooling and continuity in academic training. 
Another policy that occurred between 2003 and 20015 that increased the 
country’s school performance by about 40% was the schooling of mothers, who 
completed secondary education (Equivalent to High School). 


According to Gomes Ferreira (2017) of CEIS20),°° if we have 
more educated and encouraged mothers today, it is natural that we have 
children who are better able to insert themselves in school, get involved and 
evolve in school. It also influenced infant mortality up to 5 years of age in 
Portugal has fallen 94% since the 1970s, according to UNICEF, from 68 child 
deaths per thousand birthsto 4 per thousand in 2015. 


Still on education policies, changes were introduced by laws bringing 
autonomy and flexibility in curriculum and method, this change implemented in the 
educational system reduced in % of the teaching time in all schools in Portugal. 
According to OECD (2017) the latesi Education at a Glance, the salary is 
compatible with te labor market and top of the career, even above the average of 


other contries. 





5° The Centre for Interdisciplinary Studies of the 20th Century (CEIS20) is a research unit of the University of 
Coimbra, linked to the Institute for Interdisciplinary Research, and funded since 1998 by the Foundation for 

Science and Technology (FCT). CEIS20, with its 257 researchers, of which 109 "Integrated", distributed among 

seven Thematic Groups and two Interdisciplinary Nuclei, presents a scientific productivity based on the 

dynamics of these Groups (and Nuclei), in the framework of an interdisciplinary consciousness, understood, 

in its field of action, as a methodological resource for the knowledge and theoretical innovation 

required by the object of its research. http://www.ceis20.pt/publicacoes/ 
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Changes in the Portuguese Educational System Author of the Citation Year 
Internal Evaluation System (Formative) Fernandes 2007 
| Greater Investmentin Basic Education (1st, 2nd and 3rd | Gomes Ferreira- CEIS20 | 2000 
Cycles) 
| Systemic Work between Society and School (PISA) —_| Fernandes 2007 
Teacher Training (Bachelor and Master) Pintassilgo & Oliveira 2013 
| Education of the Mothers of Students (2003 - 2015) | Gomes Ferreira- CEIS20 | 2017 
Autonomy and Flexibility in Curriculum and Method | Legislation - Eurydice 2017 
Teacher salaries above country average | Education at a Glance 2017 
Frequent use of student retention and disapproval OECD, 2003, 2006 and | 2017 | 
2009 
New School Movement (Reading 40's) School, Catarino & Aires 2018 
Quality of Education (Teacher Selection) | John Nordahl from OECD | 2018 

















Box 24 - Changes Occurring in the Portuguese Educational System 
Source: Developed for this study 

Some of the main changes were listed in table 23 and are thus 
considered as indicators that changed PISA scores. These are structural changes 
that took years to be implemented in the academic community, Gomes Ferreira 
CEIS20 (2000) points out that the relationship between school action and 
educational development directly influenced Portugal’s score in the International 
Student Assessment Program. 

Unicef ° (2017) highlights in its report that only 15 countries among 
them Portugal has basic educational policies aimed at reducing infant mortality 
and also aimed at the parents of children, being in the most critical period of their 
development 2 to 3 years free pre-primary, being pause in breastfeeding in 
companies and extended maternity leave. Gomes Ferreira (2000 p. 69) 
emphasizes that, "the development of children in the first years of life can be as 
or more relevant to these results than one method or another applied toschool. 


Gomes Ferreira & Mota (2009) of GEIS20 states that the countries 
with the best educational results are those that are more socially just, in a study 
by the Research Group of the University of Coimbra states that societies with less 
violence, more equality, more quality of life and well-being are better evaluated 
on PISA/OECD. 


70 "World Situation of Children 2017: children in the digital world (The State of the World's Children 
2017: Children in a digital world). https://www.unicef.pt/global-pages/footer/relatorio-anual-unicef/ 
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10 FRAGILITIES EVIDENCED IN METRICS 


A partir da reuniao do material tedrico, e as avaliagdes externas 
internacionais promovidas pela International Association for the Evaluation of 
Educational Achievement - IEA (por exemplo, Reading Literacy; Third 
International Mathematics and Science Study - TIMSS), pelo Educational Testing 
Service - ETS (Second International Assessment of Educational Progress - 
SIAEP) e pela Organization for Economic CoOperation and Development - 
OECD ( International Adult Literacy Survey - IALS); (Programme for International 
Student Assessment - PISA relatorios publicados dos anos 2000, 2003, 2006, 
2009, 2012, 2015, 2018) destacamos algumas dificuldades métricas 
evidenciadas por alguns autores como: Goldstein, 1996; 2004; Kellaghan, 2003; 
Riley; Torrance, 2003 nos sistemas educacionais avaliados: 


a) the difficulty in designing a methodology that allows 
for adequate evaluation of results from a great diversity 
of curricula and contexts; b) the difficulty in guaranteeing 
the equivalence of the populations and the samples of 
students to be compared; c) the difficulty in guaranteeing 
that the translation of the tests and other instruments 
used have the same meaning in all the participating 
countries; d) the difficulty in guaranteeing that the 
subjects to which the test items refer have the same type 
of curricular relevance in the different countries; e) the 
difficulty in ensuring that all countries follow exactly the 
methodological procedures defined by the study 
coordinators regarding population identification, sample 
selection, test administration and correction or 
translation; and f) the difficulty inherent in the restrictive 
nature of data modeling and analysis, its interpretation 
and the absence of a longitudinal dimension. These and 
other problems of international studies have been 
mentioned by a diversity of researchers (FERNANDES, 
2007, p. 504). 


Perceiving the context presented as weaknesses to parameterize 
the large-scale evaluations, described above by Fernandes (2007), we analyze 
the quali-quantitative aspect described in table 25 of this work the scoring scale, 
the interdecil by means of indicators. 

The reflection on the parameters obtained in the standard deviation 
as a measure that expresses the degree of dispersion of a data set. That is, the 
standard deviation indicates how uniform a data set is. The closer the standard 
deviation is to 0, the more homogeneous the data are, in table 14 shown in figure 
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25 we have a set of estimates of the means, expressed in international 
evaluations. 

Pointed out by Klein (2011) as a problem of comparability in results 
mainly in the International Student Assessment Program (PISA) in many 
countries; for fixing school age and evaluating students at different stages of 


schooling. 


The problem is that there is no fixed rule for the 
application of the test in relation to the beginning of the 
school year and/or in relation to school age (age in full 
years in a given month for entry into a grade, which can 
only be the recommended age). The change in the 
month of application of the test implies a change in the 
composition of students in relation to the various grades 
and therefore a change in population, which makes them 
not comparable. And this has occurred. 

Another problem in changing the month of application is 
that it changes the "time of schooling" of students. There 
is a difference if the application is made at the beginning 
of the 1st semester or the 2nd semester of the school 
year (KLEIN, 2011, p. 719). 


Pointing out that the OECD made a major contribution to the 
evaluation process, according to Klein (2011) that points out, difficulties in 
homogenizing the results between countries and rank them in a ranking with 
scores measured in their tests. 

We developed a theorem to harmonize the results, we selected in 
this work with 6 (six) Indicators to analyze the results in the educational systems 
analyzed (Brazil, USA and Portugal) described in chart 21 and modeled in items 
(34) to (63). 


Finally, Wuttke (2007, p. 241) points out that "the OECD/PISA criteria 
of statistical significance are misleading because several sources of systematic 
distortion and uncertainties are quantitatively more important than the standard 
errors reported in official reports. This statement is included in this summary of 
the report published in German: PISA NACH PISA. Halt PISA, was es verspricht 
(PISA according to PISA. Does PISA keep what it promises?). 


Understanding that PISA’s proposal is to accurately measure the 
quality of educational systems, verify the effectiveness of teachers and the ability 
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of students to make decisions in the face of problems and challenges, promoting 
the future prosperity of society, and that the objectives contemplated are 
explained in the reports published by the OECD at each test, we do not consider 
here the metric difficulties involved in the comparability of results and the criteria 
of statistical significance described by Wuttke (2007). 


We emphasize that the International Student Assessment Program 
overcomes the limitations by assessing "which skills are considered essential for 
future life" (OECD, 1999, p. 7). Considering that the assessment of competencies 
within each of the four thematic domains is strictly a one-dimensional 
competence. "By directly testing knowledge and skills near the end of basic 
schooling, OECD/PISA examines the readiness of young people for adult life, [...] 


and the effectiveness of education systems" (OECD, 1999, p. 11). 


We also found in this thesis that studies related to the high score 
educational systems evaluated by PISA were planned with a more authoritarian 
orientation (SHIRLEY, 2017; ZHAO, 2014, 2016), imposing practices and 
procedures typical of the evaluating State, with characteristics of "Almost market" 
education. It is well known that the correlation between content and curricula 
would be extremely useful with the PISA evaluation indicating "Proficiency 
Levels" and Accontability”’ giving an approximate idea of what the competence 


differences mean, and comparisons between different evaluated populations. 


This thesis also focused on the influence of large-scale evaluations 
promoted by the IEA; ETS and OECD impacting on the educational systems of 
the countries tested. The 6 (six) indicators described in chart 21 present variables 
measured in the models of variance described in this study. According to 
Campbell (2013), PISA also influences entrepreneurship, creating confidence 
rates. 


71 The Accountability Concept linked to education and developed by Afonso (2009;2010) consists of the 
amount spent on education compared to the results achieved. Forming an index or thermometer (TA) 
https://seer.ufrgs.br/rbpae/article/view/19678 
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It is also noticeable that there is a unanimous movement of criticism 
of large-scale evaluations, mainly from OECD/PISA. Regarding the observations 
described in this study relating a comparison between these educational systems, 
we find the same difficulties, in both countries: Schooling; Performance; 
Enrollment; Investments; Teachers and student-teacher ratio. The countries with 


the best evaluations were able to manage these indicators as shown in table 23. 


The reflection on the presence of values developed from these tests 
is an extremely important point for questions involving education vs. values. When 
PISA was designed to be analyzed at one level (scale), the results of cognitive 
tests add only four "competence" values per student, reflecting the standard of 
best practices described in charts 16, 17 and 18. 


We verified the recognition of the usefulness of large-scale 
evaluations for educational policy planning’? and management, as well as for 
determining criteria for resource allocation in certain regions to improve results, 
define bonuses for teachers, establish rankings stimulating competition among 
schools, states, or even to evaluate the quality of teaching. From these tests, 
educational public policies, changes in educational management, and 
adjustments in educational processes can be implemented for new execution of 
control, reconceptualizing the power to induce others to do what the educational 


system wants. 


72 Educational policies are part of the country's social public policy group. Thus, they constitute an element 
of normatization of the State, guided by civil society, which aims to guarantee the universal right to quality 
education and the full development of the student. 
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11 WEAVING POSSIBLE FINAL CONSIDERATIONS 


The trajectory of this study had as main focus the changes occurred 
in the educational systems with the implementation of PISA, in the light of the 
conception of education as the basis of the process of building an evaluative 
culture, analyzed from the point of view of Applicability - Ability of other 


researchers to confirm the constructions of the researcher (LINCOLN and GUBA, 
1991). 

By analyzing to what extent the interrelationships between Rankings 
disclosed by the OECD, with the PISA Test and the Educational Systems 
evaluated, we have a construction of our own value, Truth Value considering 
education as the most important formative and encouraging function of the 
development of a society. 

The interactions described in this study compare the educational 
systems described in item 9.3; they describe structural changes to combat 
inequalities, and are described in table 23 as requirements that influenced the 
grade increase on PISA (Example of Portugal). 

However, when the evaluation practices of the countries involved in 
this study were analyzed, we found an excess of evaluations in Brazil (Chapter 
4) and USA (Chapter 6); while Portugal (Chapter 5) best placed and above the 
OECD average innovates in evaluations, reducing and simplifying their internal 
evaluations. 

When we compare the quali-quanti values of chart 22 in which the 
averages and score scale are translated into indicators, trustworthiness 
(described as an action satisfactorily, according to certain specifications, in a 
given time interval, under predetermined conditions) we have the complete view 
of the performance designed by PISA. 

The ideas extracted from these considerations can be explained by 
the expression Credibility One obtains credibility by "submitting (the results) to the 
approval of the builders of the multiple realities under study" (LINCOLN & GUBA, 
1991, p. 296). After the OECD/PISA test reports have been released, the quality 
of the education, in which the test was carried out and the educational system 


evaluated, is verified. 
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The study becomes important because it uses parameters and 
metrics to evaluate public educational policies, reflected in the proficiencies of 


students in PISA testing and demonstrated in OECD reports, becoming a 
‘benchmark’ standard. 


There is no way to dissociate the six indicators developed for this study 
(Schooling, Performance, Enrollment, Investment, Teachers and tudent-Teacher 
Ratio) from PISA, even though they claim that they do not evaluate according to 
national curricula or school knowledge, their results were interpreted as a valid 


measure for the quality of educational systems provided as parameters for 
students’ proficiency in PISA testing. 


There is the suggestion that comes from the considerations of this study plus 
the term: Prolonged engagemen - translated as a prolonged involvement and 
spending a lot of time to get a better understanding of values, behavior and 
metrics. The representations described here contribute to understand how 
standardized assessments on a large scale provide differentiated behavior over 
time, promoting changes in educational systems. 


We hope that this scientific research will not be kept, nor archived in the 
library, however it will be disseminated, socialized in the educational environment 
with the firm purpose of promoting knowledge to teachers, managers and 
academic leaders, whether in Basic Education or in Higher Education. Its 
publication in scientific journals and socialization at national and / or international 
congresses will be extremely relevant. 
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BETTER POLICIES FOR BETTER LIVES 


Equipping citizens with the knowledge and 

skills necessary to achieve their full potential, 

to contribute to an increasingly interconnected 
world, and to convert better skills into better lives 
needs to become a more central preoccupation 
of policy makers around the world. Fairness, 
integrity and inclusiveness in public policy thus 
all hinge on the skills of citizens. In working to 
achieve these goals, more and more countries are 
looking beyond their own borders for evidence 
of the most successful and efficient education 


policies and practices. 


PISA is not only the world’s most comprehensive 
and reliable indicator of students’ capabilities, it is 
also a powerful tool that countries and economies 
can use fo fine-tune their education policies...That 
is why the OECD produces this triennial report on 
the state of education around the globe: to share 
evidence of the best policies and practices, and 
to offer our timely and targeted support to help 
countries provide the best education possible for 


all of their students. 


Angel Gurria 
OECD Secretary-General 
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About PISA 


Up to the end of the 1990s, the OECD's comparisons 
of education outcomes were mainly based on 
measures of years of schooling, which are not reliable 
indicators of what people actually know and can do. 
The Programme for International Student Assessment 
(PISA) changed this. The idea behind PISA lay in 
testing the knowledge and skills of students directly, 
through a metric that was internationally agreed upon; 
linking that with data from students, teachers, schools 
and systems to understand performance differences; 
and then harnessing the power of collaboration to 
act on the data, both by creating shared points of 
reference and by leveraging peer pressure. 





The aim with PISA was not to create another layer 

of top-down accountability, but to help schools and 
policy makers shift from looking upward within the 
education system towards looking outward to the next 
teacher, the next school, the next country. In essence, 
PISA counts what counts, and makes that information 
available to educators and policy makers so they can 
make more informed decisions. 


The OECD countries that initiated PISA tried to 

make PISA different from traditional assessments in 
other ways too. In a world that rewards individuals 
increasingly not just for what they know, but for what 
they can do with what they know, PISA goes beyond 
assessing whether students can reproduce what they 
have learned in school. To do well in PISA, students 
have to be able to extrapolate from what they 

know, think across the boundaries of subject-matter 
disciplines, apply their knowledge creatively in novel 


situations and demonstrate effective learning strategies. 


If all we do is teach our children what we know, they 





might remember enough to follow in our footsteps; 
but if they learn how to learn, and are able to think 
for themselves, and work with others, they can go 
anywhere they want. 


Some people argued that the PISA tests are unfair, 
because they may confront students with problems they 
have not encountered in school. But then life is unfair, 
because the real test in life is not whether we can 
remember what we learned at school, but whether we 
will be able to solve problems that we can't possibly 
anticipate today. 


But the greatest strength of PISA lies in its working 
methods. Most assessments are centrally planned and 
then contracted to engineers who build them. That's 
how tests are created that are owned by a company 
— but not by the people who are needed to change 
education. PISA turned that on its head. The idea of 
PISA attracted the world’s best thinkers and mobilised 
hundreds of experts, educators and scientists from the 
participating countries to build a global assessment. 
Today, we would call that crowdsourcing; but 
whatever we call it, it created the ownership that was 
critical for success. 


In a nutshell, PISA owes its success to a collaborative 
effort between the participating countries, the national 
and international experts and institutions working within 
the framework of the PISA Consortium, and the OECD. 
Subject-matter experts, practitioners and policy makers 
from the participating countries worked tirelessly to 
build agreement on which learning outcomes are 
important to measure and how to measure them best; 
to design and validate assessment tasks that can reflect 
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those measures adequately and accurately across 
countries and cultures; and to find ways to compare 
the results meaningfully and reliably. The OECD 
co-ordinated this effort and worked with countries to 
make sense of the results and compile the reports. 


PISA 2018 was the seventh round of the international 
assessment since the programme was launched in 
2000. Every PISA test assesses students’ knowledge 
and skills in reading, mathematics and science; 

each assessment focuses on one of these subjects 
and provides a summary assessment of the other 

two. In 2018, the focus was on reading in a digital 
environment, but the design of the assessment also 
made it possible to measure trends in reading literacy 
over the past two decades. PISA 2018 defined 
reading literacy as understanding, using, evaluating, 
reflecting on and engaging with texts in order to 
achieve one’s goals, to develop one’s knowledge and 
potential, and to participate in society. PISA 2018 also 
collected extensive data on students’ attitudes and 
well-being. 


PISA 2018 assessed the cumulative outcomes of 
education and learning at a point at which most 
children are still enrolled in formal education: the 

age of 15. The 15-year-olds in the PISA sample must 
also have been enrolled in an educational institution 
at grade 7 or higher. All such students were eligible 
to sit the PISA assessment, regardless of the type of 
educational establishment in which they were enrolled 
and whether they were enrolled in full-time or part-time 
education. Not all of the students who were eligible 
to sit the PISA assessment were actually assessed. 

A two-stage sampling procedure first selected 

a representative sample of at least 150 schools, 
taking into account factors such as location (state 

or province; but also whether the school is located 

in a rural area, town or city) and level of education. 
Then, in the second stage, roughly 42 15-year-old 
students were randomly selected from each school to 
sit the assessment. Most countries assessed between 
4 000 and 8 000 students. Students selected to sit the 
PISA assessment received sampling weights so as to 
represent the entire PISA-eligible cohort. 


Over the past two decades, PISA has become the 
world’s premier yardstick for comparing quality, equity 
and efficiency in learning outcomes across countries, 
and an influential force for education reform. It has 
helped policy makers lower the cost of political action 
by backing difficult decisions with evidence — but 

it has also raised the political cost of inaction by 
exposing areas where policy and practice have been 
unsatisfactory. 
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Since 2000, PISA has shown that education systems 
can provide both high-quality instruction and equitable 
learning opportunities for all, and that they can 

support academic excellence in an environment that 
also nurtures students’ well-being. PISA shows what 
countries are doing to support their students and 
provides an opportunity for countries to learn from 
each other. This brochure summarises some of the initial 
findings from PISA 2018 and puts them into context. 
The full set of initial results can be found in PISA 2018 
Results (Volume |): What Students Know and Can Do; 
PISA 2018 Results (Volume II): Where All Students Can 
Succeed; and PISA 2018 Results (Volume III): What 
School Life Means for Students’ Lives. Three additional 
volumes of PISA 2018 Results — Are Students Smart 
about Money?; Effective Policies, Successful Schools; 
and Are Students Ready to Thrive in Global Societies? 
- will be published in 2020. 


PISA 2018: Insights and Interpretations 


Improving education 


Over ten million students represented by PISA in 

2018 were not able to complete even the most basic 
reading tasks — and these were 15-year-olds living 

in the 79 high- and middle-income countries that 
participated in the test. In many countries, the quality 
of the education a student acquires can still best be 
predicted by the student's or his or her school’s socio- 
economic background. In fact, the 10% most socio- 
economically advantaged students outperformed 
their 10% most disadvantaged counterparts in reading 
by 141 score points, on average across OECD 
countries. This adds up to the equivalent of over three 
years of schooling in the countries which were able 

to estimate learning progress across school grades, 
and this gap has essentially remained unchanged over 
the past decade. Moreover, there has also been no 
real overall improvement in the learning outcomes of 
students in OECD countries, even though expenditure 
on schooling rose by more than 15% over the past 
decade alone. 


It might be tempting to drop this report, and any further 
thought about improving education, right about now. 
Impossible to change anything as big, complex and 
entrenched in vested interests as education. 


But keep reading. Why? Because 15-year-old 
students in four provinces/municipalities of China 

- Beijing, Shanghai, Jiangsu and Zhejiang - 
outperformed their peers in all of the other 78 
participating education systems — in mathematics 

and science by a wide margin, and in reading, only 
Singapore came close (Figures 1, 2 and 3). In fact, 

the 10% most disadvantaged students in these four 
provinces showed better reading skills than those of the 








average student in OECD countries, and performed on 
a par with the 10% most advantaged students in some 
of them (Figure 4). True, these four provinces in eastern 
China are far from representing China as a whole, but 
the size of each compares to that of a typical OECD 
country, and their combined populations amount 

to over 180 million. What makes their achievement 
even more remarkable is that the level of income of 
these four Chinese regions is well below the OECD 
average. At the same time, they have a long way to go 
when it comes fo improving the social and emotional 
outcomes, and other aspects of students’ well-being 
that were measured by PISA 2018, areas where other 
countries excel (more on that later). 


Itis also noteworthy that some of today’s highest-performing 
education systems have only recently attained their 

top positions. Less than 17% of 55-65 year-old 
Singaporeans scored at level 3 or higher in literacy 

in the Survey of Adult Skills (a product of the OECD 
Programme for the International Assessment of Adult 
Competencies, a kind of PISA for adults) — one of the 
smallest proportions amongst participating countries 

— while 63% of 16-24 year-olds did so, one of the 
largest proportions. And, as noted before, in PISA 
2018, 15-year-old Singaporeans scored not statistically 
differently from the four provinces/municipalities of China 
in reading. Amongst OECD countries, Estonia has 
advanced steadily to the top, despite the fact that its 
expenditure per student remains about 30% lower than 
the OECD average (Figures 5 and 6). 


Portugal advanced to the OECD average level 
despite being severely hit by the financial crisis. 
Some countries that still perform well below the 
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Figure 1 


Reading 


Comparing countries” and economies’ 
performance in reading 


*S.D. = standard deviation 


1. Did not meet response-rate standards; further 
analyses could exclude a large bias in the published 
results due to non-response (see Annexes A2 and A4 
in OECD (2019), PISA 2018 Results (Volume |): What 
Students Know and Can Do, PISA, OECD Publishing, 
Paris; https: //doi.org/ 10.1787 /5107c754-en). 


Source : OECD, PISA 2018 
Database, Table |.B 1.4; Figure 1.4.1. 
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Comparing countries’ and economies’ 
performance in mathematics 
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*S.D. = standard deviation 


1. Data did not meet the PISA technical standards but 
were accepted as largely comparable (see Annexes 
A2 and A4 in OECD (2019), PISA 2018 Results 


(Volume I): What Students Know and Can Do, PISA, 
OECD Publishing, Paris; 


https://doi.org/ 10.1787 /5{07c754-en. 


Source : OECD, PISA 2018 
Database, Table |.B1.5; Figure 1.4.2. 
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Science 


Comparing countries’ and economies’ 
performance in science 


*S.D. = standard deviation 


1. Data did not meet the PISA technical standards but 
were accepted as largely comparable (see Annexes 
A2 and A4 in OECD (2019), PISA 2018 Results 
(Volume I}: What Students Know and Can Do, PISA, 
OECD Publishing, Paris; 
https://doi.org/ 10.1787 /5{07c754-en. 


Source : OECD, PISA 2018 
Database, Table |.B 1.6; Figure 1.4.3. 
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Figure 4 © Mean performance in reading, by international decile of socio-economic status 
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Note: Bottom, second, ninth and top deciles correspond to the average performance of students who are in the corresponding deciles of the distribution of the PISA index of economic, 
social and cultural status across all countries and economies; the middle decile corresponds to students whose socio-economic status ranges from the 45th to the 55th percentile of this 
distribution. 

Only results of countries and economies with at least 3% of students in each international decile are shown. 

Countries and economies are ranked in ascending order of the mean reading performance of students in the international middle decile of socio-economic status. 


Source: OECD, PISA 2018 Database, Table IIB 1.2.2; Figure II.2.2. 
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OECD average saw remarkable improvements in 

their students’ performance, most notably Albania, 

the Republic of Moldova, Peru and Qatar. Turkey's 
improvement between 2003 and 2018 may look 
somewhat less impressive, but Turkey was able to 
double the coverage of the 15-year-olds who are 
enrolled in school and covered by PISA from 36% to 
73% during that period. Five other countries — namely 
Albania, Brazil, Indonesia, Mexico and Uruguay 

— also significantly increased enrolment rates in 
secondary education over their participation in PISA 
and maintained or improved their mean reading, 
mathematics and science performance. This shows that 
the quality of education does not have to be sacrificed 
when increasing access to schooling. 


Some countries were able to move to a more positive 
trajectory in recent years after a period of stagnation 
or decline. Sweden showed an improving trend in all 
three subjects between 2012 and 2018, reversing 
earlier declines in mean performance. Argentina, the 
Czech Republic and Ireland saw recent improvements 
in reading; Denmark, Ireland, Jordan, Slovenia and 
the United Kingdom in mathematics; and Jordan and 
Montenegro in science. In some countries, some 

of these trends can be related to changes in the 
demographic composition of the student body; but in 
no country do such demographic changes alter the 
picture dramatically. 


PISA also shows that in most countries excellence 

in education is apparent amongst some of the most 
disadvantaged students and schools. On average 
across OECD countries, one in ten disadvantaged 
students was able to score in the top quarter of 
reading performance in his or her country, indicating 
that disadvantage is not destiny. In fact, in Australia, 
Canada, Estonia, Ireland and the United Kingdom, all 
of which scored above the OECD average, more than 
13% of disadvantaged students were academically 
resilient. Similarly, more than 30% of immigrant students 
in Brunei Darussalam, Jordan, Panama, Qatar, Saudi 
Arabia and the United Arab Emirates scored in the top 
quarter of reading performance. These successes do 
not come about by chance. Factors that PISA shows 
to be positively associated with academic resilience 
include support from parents, a positive school climate 
and having a growth mindset (see more on this later). 


In the same way as social disadvantage does not 
automatically lead to poor educational performance 
for students and schools, the world is no longer 
divided between rich and well-educated nations and 
poor and badly educated ones. When comparing 
countries that score similarly in PISA, their income 
levels vary widely. History shows that countries with 
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the determination to build a first-class education 
system can achieve this even in adverse economic 
circumstances, and their schools today will be their 
economy and society tomorrow. So it can be done. 


And it must be done. Without the right education, 
people will languish on the margins of society, 
countries will not be able to benefit from technological 
advances, and those advances will not translate into 
social progress. It will not be possible to develop fair 
and inclusive policies and engage all citizens if a lack 
of education prevents people from fully participating in 
society. 


But change can be an uphill struggle. Young people 

are less likely to invest their time and energy in better 
education if that education seems irrelevant to the 
demands of the “real” world. Businesses are less likely 

to invest in their employees’ lifelong learning if those 
workers might move away for a better job. Policy makers 
often prioritise the urgent over the important — even if 

the latter includes education, an investment in the future 
well-being of society. 


Read more about these issues in Chapters 4 
and 9 in PISA 2018 Results (Volume |): What 
Students Know and Can Do. 
https://doi.org/10.1787/5f07c754-en 


And in Chapter 2 in PISA 2018 Results 
{Volume II): Where All Students Can Succeed. 
https://doi.org/ 10.1787/b5fd1b8f-en 
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Figure 5 * Curvilinear trajectories of average performance in reading across PISA assessments 
Direction and trajectory of trend in mean performance 
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Notes: Figures are for illustrative purposes only. Countries and economies are grouped according to the overall direction of their trend (the sign and significance of the average three- 
year trend) and to the rate of change in the direction of their trend (the sign and significance of the curvature in the estimate of quadratic trends). 

Only countries and economies with data from at least five PISA reading assessments are included. Not all countries and economies can compare their students’ performance over the 
same period. For each country/economy, the base year, starting from which reading results can be compared, is indicated in parentheses next to the country’s/economy’s name ("00" = 
2000, “01” = 2001, etc.). Both the overall direction and the change in the direction may be affected by the period considered. 

OECD average-23 refers to the average of all OECD countries with valid data in all seven assessments; Austria, Chile, Estonia, Israel, Luxembourg, the Netherlands, the Slovak Republic, 
Slovenia, Spain, Turkey, the United Kingdom and the United States are not included in this average. 


Source: OECD, PISA 2018 Database, Table |.B 1.10; Figure 1.9.1. 
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Getting ready tor the digital world 


While people have different views on the role that 
digital technology can and should play in schools, we 
cannot ignore how digital tools have so fundamentally 
transformed the world outside of school. Everywhere, 
digital technologies are offering firms new business 
models and opportunities to enter markets and 
transform their production processes. They can make 
us live longer and healthier, help us with boring or 
dangerous tasks, and allow us to travel into virtual 
worlds. People who cannot navigate through the 
digital landscape can no longer participate fully in our 
social, economic and cultural life. 


PISA shows how access to new technologies has 
increased at a remarkable rate. In the 2009 PISA 
assessment, about 15% of students in OECD countries, 
on average, reported that they did not have access 

to the Internet at home. By 2018, that proportion had 
shrunk fo less than 5%. The growth in access to online 
services is likely to be even steeper than suggested by 
these percentages, which hide the improvements in the 
quality of Internet services and the explosion of mobile 
Internet access over the past decade. 


Furthermore, in all countries that distributed an optional 
questionnaire on students’ familiarity with these 
technologies as part of PISA 2018, the amount of time 
that 15-year-old students in OECD countries spent 

on line outside of school increased between 2012 
and 2018 — by an average of more than 1 hour per 
day (on both weekdays and weekends). Students 
now spend about 3 hours on line outside of school on 
weekdays, on average, and almost 3.5 hours on line 





on weekend days. For young people, the digital world 
is becoming a sizeable part of the real world. 


While improved access to new technologies provides 
unprecedented opportunities, it also raises the bar 
of what it means to be proficient in reading. Students 
growing up with a great smartphone but a poor 
education will face real risks. The smartphone has 
transformed the ways in which people read and 
exchange information, and digitalisation has resulted 
in the emergence of new forms of text, ranging 

from the concise (text messages; annotated search- 
engine results) to the lengthy and unwieldy (tabbed, 
multipage websites or complex archival material). In 
the past, students could find clear and often singular 
answers to their questions in carefully curated and 
government-approved textbooks, and they could 
generally trust those answers to be true. Today, they 
will find hundreds of thousands of answers to their 
questions on line, and it is up to them to figure out 
what is true and what is false, what is right and what is 
wrong. While in many offline situations readers can 
assume that the author of the text they are reading 

is competent, well-informed and benevolent, when 
reading online blogs, forums or news sites readers 
must constantly assess the quality and reliability of the 
information, based on implicit or explicit cues related 
to the content, format or source of the text. 


This is not exactly a new phenomenon, but the speed, 
volume and reach of information flows in the current 
digital ecosystem have created the perfect conditions 
for fake news to thrive, affecting public opinion and 
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political choices. In this “post-truth” climate, quantity 
seems to be valued more than quality when it comes 
to information. Assertions that "feel right” but have no 
basis in fact become accepted as truth. Algorithms 

hat sort people into groups of like-minded individuals 
create social media echo chambers that amplify views, 
and leave individuals uninformed of and insulated 

rom opposing arguments that may alter their beliefs. 
There is a scarcity of attention, but an abundance of 
information. 


The more knowledge that technology allows students 





o search and access, the more important becomes 
deep understanding and the capacity to make 
sense of content. Understanding involves knowledge 
and information, concepts and ideas, practical 

skills and intuition. But fundamentally it involves 
integrating and applying all of these in ways that 
are appropriate to the learner's context. Reading is 
no longer mainly about extracting information, it is 
about constructing knowledge, thinking critically and 
making well-founded judgements. Contrast this with 
the findings from this latest round of PISA, which show 
that fewer than 1 in 10 students in OECD countries 
was able to distinguish between fact and opinion, 
based on implicit cues pertaining to the content or 
source of the information. Education has won the race 
with technology throughout history, but there is no 
guarantee that it will do so in the future. 


The PISA assessments have evolved to better 

capture these demands. In the 2018 assessment, 

the description of what top-performing students are 
able to do in reading included not only being able to 
understand and communicate complex information, 
but also the capacity to distinguish between fact and 
opinion when reading about an unfamiliar topic. The 
nature of texts and the type of problems included 

in the PISA 2018 assessment of reading reflected 

the evolving nature of reading in increasingly digital 
societies. Specifically, the 2018 reading assessment 
placed greater emphasis on the ability to find, 
compare, contrast and integrate information across 
multiple sources. In order fo assess multiple-source 
reading, new assessment tasks were designed, 
based on texts composed of several smaller units, 
each created by a different author or authors or at 
different times. Examples of these kinds of texts are an 
online forum with multiple posts and a blog that links 
to a newspaper article. Computer delivery made it 
possible to use various digital navigation tools, such as 
hyperlinks or tabs, and to present such tasks in realistic 
scenarios, in which the amount of available text 
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sources increases as the student progresses through the 
assessment. (To see what some of these tasks were like, 
go to www.oecd.org/pisa/test/) 


The results from the PISA 2018 assessment suggest 
that improvements in education have not kept up with 
these rising demands. The proportion of 15-year-old 
students who scored at the highest levels rose only 
marginally across OECD countries, from 7% in 2009 
to 9% in 2018. These students, who attained Level 5 or 
6 in the PISA reading test, were able to comprehend 
lengthy texts, deal with concepts that are abstract or 
counterintuitive, and establish distinctions between 

fact and opinion, based on implicit cues pertaining 

to the content or source of the information. Even in 
Singapore, the country with the largest share of top 
performers, only one in four 15-year-old students 

was able to reach this level. In the four participating 
Chinese provinces/municipalities, Canada, Finland 
and Hong Kong (China), at least one in seven students 
were able to do so. 


Beyond the requisite knowledge and skills, PISA also 
shows that students seem to read less for leisure and to 
read fewer books of fiction, magazines or newspapers 
because they want to (as opposed to because they 
have to). Instead, they read more to fulfil practical 
needs, and they read more in online formats, such as 
chats, online news or websites containing practical 
information. In 2018, more students considered 
reading “a waste of time” (+5 percentage points, on 
average across OECD countries) and fewer students 
read for enjoyment (-5 percentage points) than their 
counterparts did in 2009. 


Humans were always better at inventing new tools than 
using them wisely, but as the influence that schools — 
and families — have over what students read declines, 
it is essential that schools redouble their efforts to 
promote reading proficiency to meet the demands of 
he digitalised world. All students need to be able to 
read complex texts, distinguish between credible and 
untrustworthy sources of information, and between 

act and fiction, and question or seek to improve the 
accepted knowledge and practices of our times. 


Beyond that, in a world shaped by artificial 





intelligence, education is no longer just about teaching 
people something, but about helping people build a 
reliable compass and the navigation tools to find their 
own way through an increasingly volatile, uncertain 





and ambiguous world. Tomorrow's schools will 
need to help students think for themselves and join 
others, with empathy, in work and citizenship. They 


will need to help students develop a strong sense of 
right and wrong, a sensitivity to the claims that others 
make on them, and a grasp of the limits on individual 
and collective action. At work, at home and in the 
community, people will need a deep understanding of 
how others live, in different cultures and traditions, and 
how others think, whether as scientists or artists. The 
PISA 2018 assessment of global competence explored 
some of these capacities. Results from that assessment 


will be published in 2020. 


Read more about these issues in Chapters | 
and 5 in PISA 2018 Results (Volume |): What 
Students Know and Can Do. 
https://doi.org/10.1787/5f07c754-en 
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Figure 6 * Students’ proficiency in reading (6a: computer-based assessment/6b: paper-based assessment) 
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Figure 6b Students’ proficiency in reading (paper-based assessment) 
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Note: Countries and economies are ranked in descending order of the percentage of students who performed at or above Level 2. 


Source: OECD, PISA 2018 Database, Tables I.B1.1 and I.A2.1; Figures 1.5.1 and 1.5.2. 
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Building strong foundations 


The rising bar of success in education in the digital 
age puts even greater pressure on education systems 
to secure strong foundations. There is a great risk that 
technology will super-empower those with strong 
knowledge and skills while leaving those with weak 
foundations further behind. 


Only 77% of students, on average across OECD 
countries, attained Level 2 proficiency in reading 
(Figure 6). Level 2 marks the point at which students 
have acquired the technical skills to read, and can use 
reading for learning. At a minimum, these students are 
able to identity the main idea in a text of moderate 
length, find information based on explicit criteria, and 
reflect on the purpose and form of texts when explicitly 
directed to do so. 


The share of 15-year-old students, in grade 7 and 
above, who reached this basic level of proficiency 
in reading ranged trom close to 90% in the four 
provinces/municipalities of China, Estonia, Macao 
(China) and Singapore, to less than 10% in Cambodia, 
Senegal and Zambia (countries that participated in 
the PISA for Development assessment in 2017). The 
share of 15-year-old students who attained minimum 
levels of proficiency in mathematics (at least Level 

2) varied even more — between 98% in Beijing, 
Shanghai, Jiangsu and Zhejiang (China) and 2% in 
Zambia. These numbers show that all countries still 
have some way to go towards reaching the global 
goals for quality education, as defined in the UN 
Sustainable Development Goal for education, and 
for many countries it remains a long way. Some 78% 
of students attained Level 2 or higher in science, on 





average across OECD countries. These students can 
recognise the correct explanation for familiar scientific 
phenomena and can use such knowledge to identity, 
in simple cases, whether a conclusion is valid based on 
the data provided. 


In some education systems low performers are spread 
across many different schools, while in others, low 
performers tend to be clustered in certain schools, 
often compounded with social disadvantage. In 
some of these countries the between-school variation 
in performance is the result of stratification and 
selection, and thus an in-built feature of the school 
system. In systems where low performers are more 
often concentrated in specific schools or types of 
schools, such as Germany, Hungary, Israel, Lebanon, 
the Netherlands, the Slovak Republic and Turkey, it is 
important to ensure that especially those schools with 
low performance receive adequate resources and 
support. 


Interventions can also be targeted at socio-economically 
disadvantaged students and/or schools. In almost 

all countries that participated in PISA 2018, students 
who were disadvantaged compared with their peers 
in their country were less likely to attain the minimum 
level of proficiency in reading. However, the strength 
of the relationship between a student's socio-economic 
status and his or her performance varied greatly 
across countries and economies. In systems where the 
relationship between the two was particularly strong, 
including Belarus, France, Hungary, Luxembourg, Peru, 
the Philippines, Romania and the Slovak Republic, 
interventions targeting disadvantaged students 
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would be particularly important. Since in Belarus, 
Hungary, Peru, the Philippines and the Slovak Republic 
disadvantaged students were clustered in certain 
schools, interventions targeting these schools would be 
most appropriate. 


In many countries, immigrant students tended to be less 
likely than their native-born peers fo attain the minimum 
level of proficiency in reading. Targeted support for 
immigrant students would seem most appropriate in 
Belgium, Denmark, Finland, Germany, Iceland, the 
Netherlands, Slovenia and Sweden, where at least 
5% of students have an immigrant background. These 
students are at least three times as likely as students 
without an immigrant background to score below the 
minimum level of proficiency in reading. 


Boys, especially disadvantaged boys, also need 
special support if they are to improve their reading 
performance. In all but 3 PISA-participating countries 
and economies (the exceptions are the 4 PISA- 
participating provinces/municipalities of China, Estonia 
and Macao [China]) at least 20% of disadvantaged 
boys did not attain the minimum level of proficiency in 
reading. In 24 countries and economies, more than 
70% of disadvantaged boys scored below that level. 


Perhaps most worryingly, the proportion of low 
performers, both girls and boys, increased between 
2009 and 2018, on average across OECD 
countries. While in all PISA-participating countries 
and economies girls outperformed boys in reading in 
2018, in 20 countries and economies girls’ reading 
performance declined over the past decade. Iceland, 
Italy, Japan and New Zealand, in particular, need 
to monitor this decline closely, even though boys’ 
performance remained stable in these countries over 
the period. 


Evidence from PISA shows that boys and girls can 
improve their performance. In the Czech Republic, 
Denmark, Estonia, Ireland, Macao (China), Singapore, 
Slovenia, the United Kingdom and the United States, 
the proportion of top performers in reading increased 
amongst both boys and girls, while the proportion of 
low performers in reading remained stable or shrank. 
The Czech Republic, Denmark, Macao (China), 
Singapore and Slovenia also saw a narrowing of 
the gender gap in mathematics performance while 
achieving at high levels. 


Some worry that if efforts and resources are directed 
towards low-performing students, high-performing 
students will suffer. However, PISA results show that 
countries can pull up low performers without adversely 
atfecting other students. For example, 22 countries 
and economies saw improvements in the average 
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reading score amongst the lowest-performing 10% 
of students; none of them saw a decline in average 
performance and 14 saw improvements. In only two 
of these 22 countries and economies did the scores 
of the highest-performing 10% of students actually 
decline. 


Read more about these issues in Chapters 
4-10 in PISA 2018 Results (Volume |): What 
Students Know and Can Do. 
https://doi.org/10.1787/5f07c754-en 


And in Chapters 2, 4, 7,9 and 10 

in PISA 2018 Results (Volume II): Where All 
Students Can Succeed. 

https://doi.org/ 10.1787/b5fd1b8f-en 
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Reconciling equity and excellence 


For those with the right knowledge and skills, 
digitalisation and globalisation have been liberating 
and exciting; but for those who are insufficiently 
prepared, they often mean vulnerable and insecure 
work, and a life with few prospects. Economies are 
shifting towards regional hubs of production that are 
linked together by global chains of information and 
goods, but that are concentrated where comparative 
advantage can be built and renewed. This makes the 
distribution of knowledge and wealth crucial, and that 
is linked to the distribution of education opportunities. 


For a start, PISA shows that the performance gap 
between top-performing and low-achieving students 
varies widely across education systems. The largest 
gaps were found in Israel, Lebanon, Malta and the 
United Arab Emirates, meaning that learning outcomes 
at age 15 in these countries are highly unequal. Not 
surprisingly, the smallest ditferences between high 

and low-achieving students tended to be observed 
amongst countries and economies with the lowest 
mean scores. In Kosovo, Morocco and the Philippines, 
even the highest-performing students scored only 
around the OECD average. Countries also differed in 
the performance variation that lay between schools. 

In Finland, less than 7% of that variation lay between 


schools, so the closest school is always the best school. 


In Israel, that proportion was 78% (Figure 7). 


Equity and socio-economic status 


However, equity does not mean that all students 
have equal outcomes in every subject; rather it means 
that whatever variations there may be in education 
outcomes, these should not be related to students’ 





background, including socio-economic status, gender 
or immigrant background. In other words, there can be 
numerous reasons why some students perform better 
than others, but the performance differences should not 
be the result of the conditions in which students learn. 
Therefore, PISA measures equity by the extent to which 
education outcomes, such as student performance, 
students’ attitudes and beliefs, and students’ 
expectations for their future, are related to students' 
personal background. The weaker the relationship, the 
more equitable PISA considers a school system fo be. 


The motivation for that approach to defining equity 

is simple: children from wealthier families may find 
many open doors to a successtul life, but children from 
poor families often have just one chance in life - and 
that is a good teacher and school that give them an 
opportunity to develop their potential. Those who 

miss that boat rarely catch up, and OECD data show 
that subsequent education opportunities in life tend to 
reinforce early education outcomes. 


In France, Germany, Hungary, Israel, Peru and the 
Slovak Republic, the gap in reading performance 
between the 10% most socio-economically 
advantaged and the 10% most disadvantaged 
students was over 170 score points — the equivalent of 
well over four years of schooling in the countries that 
were able to estimate progress across school grades. 
Of course, where there are students with economic or 
social advantages, it is likely that they will be better 
equipped to do well. This is not just about poverty 

of material resources, but equally about poverty of 
aspirations and hope. However, and as noted before, 
in some countries even the most disadvantaged 
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students do well by international standards, which 
shows that this relationship is not inevitable. 


More generally, PISA shows that the impact of 
social background on success in education varies 
greatly across countries. The most impressive 
outcome of world-class school systems is that they 
deliver high-quality education across the entire 
system. In Australia, Canada, Denmark, Estonia, 
Finland, Hong Kong (China), Japan, Korea, Macao 
(China), Norway and the United Kingdom, for 
example, average reading performance was higher 
than the OECD average while the relationship 
between socio-economic status and reading 
performance was weaker than the OECD average. 


There seems to be no association between trends in 
mean performance and a widening or narrowing of 
performance gaps. Some countries improved in PISA 
mainly as a result of low-achieving students catching up to 
higher-pertorming students; others improved by nurturing 
high performance amongst their top-performing students; 
and many improved by helping all students succeed at 
higher levels. 


The issue is more pertinent for education policy when 
it comes to the interplay between a student's and 

a school’s social background and how these are 
related to learning outcomes. In many countries, the 
school's socio-economic context influences the kind 

of education children are acquiring, and the quality 

of schooling can shape the socio-economic contexts 
of schools. If schools are popular, house prices in 

their catchment areas can rise, further segregating the 
population. People with fewer assets, lower income 
and less education end up finding housing where 
education and social opportunities are poorer. The 
result is that in most countries, differences in education 
outcomes related to social inequalities are stubbomly 
persistent, and too much talent remains latent. Although 
private schools tend to be more selective, which 
contributes to social segregation in the school system, 
in many countries most of the social segregation across 
schools comes from within the public sector rather than 
from social segregation between public and private 
schools (Figure 8). 


Some countries still have a long way to go in 
moderating between-school differences. In Argentina, 
Bulgaria, the Czech Republic, Hungary, Peru, the 
Slovak Republic and the United Arab Emirates, a typical 
disadvantaged student has only a one-in-eight chance 
of attending the same school as high achievers (those 
who score in the top quarter of reading performance 
in PISA). By contrast, in 14 countries, including the 
OECD countries Canada, Denmark, Estonia, Finland, 
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Iceland, Ireland, Norway, Portugal and Sweden, 
disadvantaged students have at least a one-in-five 
chance of having high-achieving schoolmates. 


High levels of social and ability stratification 

between schools can have an impact on the learning 
opportunities available to students and thus on 
education outcomes. Limited social diversity in schools 
implies that disadvantaged students are more likely 

to be enrolled in schools that have disproportionately 
large concentrations of low achievers — which, 

may negatively affect their performance. Unless 
disadvantaged schools are allocated sufficient 
resources to compensate for their shortfalls, social and 
academic segregation between schools may widen 
the gaps in outcomes related to socio-economic status. 


The good news is that improving education is not all 
about the volume of resources. PISA results show that 
there is a positive relationship between investment 

in education and average performance — up toa 
threshold of USD 50 000 in cumulative expenditure 
per student from age 6 to 15 (Figure 9). However, 
after that threshold, there is almost no relationship 
between the amount invested in education and student 
performance. For example, Estonia and Latvia invest 
similarly in primary and lower secondary education 
(a cumulative expenditure of about USD 65 O00 
per student), yet Estonia scored more than AO points 
above Latvia in reading. In turn, Australia, the United 
Kingdom and the United States all spend more than 
USD 107 000 per student from age 6 to 15, yet 
scored no better than (and in some cases, below) 
Canada, Ireland and New Zealand, all of which 
spend between 10% and 30% less. The results are 
similar when it comes to the relationship between 
spending per student and the impact of social 
background on reading performance, i.e. the countries 
spending more do not necessarily show a weaker 
relationship between students’ socio-economic status 
and their performance. 





What may matter more after a threshold is reached 

is how resources are allocated. The picture is similar 
when comparing the learning time that students invest 
(Figure 10). In Finland, the country where students 
spend the least time learning, student performance 

is comparatively high, whereas in the United Arab 
Emirates, the country with the longest study hours, 
learning outcomes are comparatively poor. The lack of 
a correlation between the amount of learning time and 
learning outcomes illustrates that learning outcomes 
are always the product of the quantity of learning 

time, and the quality of learning and the instructional 
environment. 
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Figure 7 * Variation in reading performance between and within schools 
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Note: All analyses are restricted to schools with the modal ISCED level for 15-year-old students. 
Countries and economies are ranked in descending order of the between-school variation in reading performance, as a percentage of the total variation in performance across OECD 
countries. 


Source: OECD, PISA 2018 Database, Table II.B 1.4.1; Figure 11.4.1. 
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Figure 8 * Segregation of advantaged and disadvantaged students 
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Notes: All analyses are restricted to schools with the modal ISCED level for 15-year-old students. 
The isolation index measures whether students of type A are more concentrated in some schools. The index is related to the likelihood of a representative type A student to be enrolled in 


schools that enrol students of another type. It ranges from O to 


1, with O corresponding to no segregation and 


1 to full segregation. 


A socio-economically advantaged student is a student in the top quarter of the PISA index of economic, social and cultural status (ESCS) in his or her own country/economy. 
A socio-economically disadvantaged student is a student in the bottom quarter of the PISA index of economic, social and cultural status (ESCS) in his or her own country/economy. 


Source: OECD, PISA 2018 Database, Table II.B 1.4.7; Figure 1I.4.5. 


Many education systems can do better in aligning 
resources with needs and moderate social inequality 
amongst schools. When it comes to the allocation 
of material resources, much progress has been 
achieved. According to PISA, some school systems 
succeed in providing sufficient material and staff 
resources to all schools, including disadvantaged 
schools. In Bulgaria, Denmark, Latvia, Norway and 
Poland, instruction appeared not to be hindered by 
shortages of educational material or staff, according 
to school principals, and there was no significant 
difference in these reports between principals of 
advantaged schools and those of disadvantaged 
schools (Figure 11). Many countries have introduced 
formula-based approaches to funding whereby the 
resources allocated to a school depend on its socio- 
economic context. 


However, attracting the most qualified teachers to 

the most challenging classrooms remains a major 
challenge for most countries. For example, PISA results 
show that in several countries, more teachers are 
allocated to disadvantaged schools than advantaged 
schools, but these teachers tend to be less experienced 
and hold lower qualifications. On average across 
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OECD countries, 40% of teachers in disadvantaged 
schools but 48% of teachers in advantaged schools 
had at least a master's degree; and in 42 countries 
and economies, principals of disadvantaged schools 
were significantly more likely than those of advantaged 
schools to report that their school’s capacity to provide 
instruction was hindered by a shortage of education 
staff. PISA also found that in 7 of the 19 countries and 
economies that distributed an optional questionnaire 
for teachers, the proportion of teachers with less than 
five years of experience was larger in disadvantaged 
schools than in advantaged schools. Students then 
face a double disadvantage: one that comes from their 
home background and another that is created by the 
school system (Figure 12). 


OECD analyses show that it is not as simple as paying 
teachers who work in disadvantaged schools more; 

it requires holistic approaches in which teachers feel 
supported in their professional and personal lives 
when they take on additional challenges, and when 
they know that additional effort will be valued and 
publicly recognised. Some education systems have 
been moving in this direction. Singapore sends its best 
teachers to work with the students who are having the 
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Figure 9 * Reading performance and spending on education 
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Source: OECD, PISA 2018 Database, Tables |.B1.4 and Figure; 1.4.4. 


greatest difficulty meeting Singapore's high standards. 
In Japan, officials in prefectural offices will transfer 
effective teachers to schools with weak faculties to make 
sure that all students have equally capable instructors. 
Shanghai has established a system of financial transfer 
payments to schools serving disadvantaged students 
and career structures that incentivise high-performing 
teachers fo teach in disadvantaged schools. Shanghai 
also pairs high-performing districts and schools with 
low-performing districts and schools, so that the 
authorities in each can exchange and discuss their 
development plans with each other, and institutes for 
teachers’ professional development can share their 
curricula, teaching materials and good practices. The 
government commissions “strong” public schools to 
take over the administration of “weak" ones by having 
the “strong” school appoint one of its experienced 
leaders, such as the deputy principal, to be the 
principal of the “weak” school, and sending a team 
of experienced teachers to lead in teaching. The 





Cumulative expenditure per student over the theoretical duration of studies (in US do 


underlying expectation is that the ethos, management 
style and teaching methods of the high-performing 
school can be transferred to the poorer-performing 
school. 


In the state of Ceard, in Brazil, the highest-performing 
schools receive a significant reward in additional 
financial resources that allows them to hire more 
specialised teachers and experts. They do not use 
these additional resources in their own school; but are 
required to allocate them to the schools that struggle 
most. So everyone wins: the high-performing schools 
gain additional prestige and an expanded team, and 
the low-performing schools benefit from the expertise 
of high-performing schools — which might have been 
more valuable to them than additional money. 


All this being said, it is often difficult for teachers to 
allocate scarce additional time and resources to the 
children with the greatest needs. People who laud 
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Figure 10 * Reading performance and total learning time per 
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Source: OECD, PISA 2018 Database, Table |.B1.4; and OECD, PISA 2015 Database, Figure II.6.23; and PISA 2018 Figure |.4.5. 


the value of diversity in classrooms are often talking 
about the classes other people's children attend. 

It is challenging to convince socio-economically 
advantaged parents whose children go to school with 
other privileged children that everyone is better off 
when classes are socially diverse. Policy makers, too, 
find it hard to allocate resources where the challenges 
are greatest and where those resources can have the 
biggest impact, often because poor children usually 
don't have someone lobbying for them. It is worth 
studying how countries that have addressed these 
challenges successfully have dealt with the political 
economy of these changes. 


In sum, all countries have excellent students, but too 
few countries have enabled alll of their students to 
excel and fulfil their potential to do so. The education 
that wise parents want for their children is what 
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public policy should strive to achieve for all children. 
Achieving greater equity in education is not only 

a social justice imperative, it is also a way to use 
resources more efficiently, increase the supply of 

skills that fuel economic growth, and promote social 
cohesion. Not least, how we treat the most vulnerable 
students and citizens shows who we are as a society. 
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Figure 11 ° Difference in shortage of educational material and staff, by schools’ socio-economic profile 
Results based on principals’ reports 
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Notes: Statistically significant differences are shown in a darker tone. 

The socio-economic profile is measured by the school’s average PISA index of economic, social and cultural status. 

For this analysis, the sample is restricted to schools with the modal ISCED level for 15-year-old students. 

Countries and economies are ranked in ascending order of the difference in the mean index of shortage of education staff. 


Source: OECD, PISA 2018 Database, Tables II.B 1.5.13 and II.B 1.5.14; Figure II.5.5. 
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Figure 12 ¢ Percentage of teachers with at least a master’s degree, by schools’ socio-economic profile 





Results based on principals’ reports 
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Notes: Statistically significant differences are shown in a darker tone. 

Education levels correspond to level 5A master's degree and level 6 of the International Standard Classification of Education (ISCED-1997). 
The socio-economic profile is measured by the school’s average PISA index of economic, social and cultural status. 

For this analysis, the sample is restricted to schools with the modal ISCED level for 15-year-old students. 
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Couniries and economies are ranked in descending order of the percentage of teachers in disadvantaged schools with at least an ISCED 5A qualification. 


OECD average-36 refers to the arithmetic mean across OECD countries (and Colombia), excluding Spain. 
Source: OECD, PISA 2018 Database, Table II.B 1.5.4; Figure II.5.1. 
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Equity and immigration 


The number of students with an immigrant background 
has grown considerably over the past 20 years in 
most OECD countries. In 2015 alone, an estimated 
4.8 million immigrants arrived in OECD countries, a 
wave that reinforced a long and steady upward trend. 
How schools and education systems respond to the 
challenges and opportunities that arise with immigrant 
flows has profound implications for the economic and 
social well-being of all members of society, including 
immigrants themselves. 


On average across OECD countries, some 13% 

of students in 2018 had an immigrant background, 

p from 10% in 2009. In most countries, immigrant 
tudents tended to be socio-economically 
sadvantaged, with the largest proportions in Austria, 
enmark, Finland, France, Germany, Greece, Iceland, 
he Netherlands, Norway, Slovenia and Sweden, 
where nearly one in two immigrant students was 
disadvantaged. 


soagec 


However, despite media-stoked concern, this growth 
in the share of students with an immigrant background 
did not lead to a decline in the education standards in 
host communities. That may be surprising, but only at 
first glance. While it is true that migrants often endure 
economic hardship and precarious living conditions, 
many immigrants bring to their host countries high 
aspirations for education, and valuable knowledge 
and skills. On average across OECD countries, the 
majority of the first-generation immigrant students who 
took part in the PISA 2015 assessment had at least one 
parent who had attended school for as many years as 
the average parent in the host country. 





Still, many children with an immigrant background face 
enormous challenges at school. They need to adjust 
quickly to different academic expectations, learn in a 
new language, forge a social identity that incorporates 
both their background and their adopted country of 
residence — and withstand conflicting pressures from 
family and peers. These difficulties are magnified when 
immigrants are segregated in poor neighbourhoods 
with disadvantaged schools. It should thus come as no 
surprise that PISA data have consistently shown, in most 
countries, a performance gap between students with 
an immigrant background and native-born students. 


In 2018, the average difference in reading 
performance across OECD countries between 
immigrant and non-immigrant students was 41 score 
points in favour of non-immigrant students. The 
ditference shrank to 24 score points after accounting 
for students’ and schools’ socio-economic profile 
(Figure 13). 
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However, the considerable cross-country variation 

in performance between immigrant students and 
students without an immigrant background is striking, 
even after accounting for socio-economic status. 

Even if the culture and the education acquired before 
migrating have an impact on student performance, the 
country where immigrant students settle seems to matter 
significantly. 


The average performance disadvantage of immigrants 
should not mask the finding that many immigrant 
students overcome considerable obstacles and excel 
academically. On average across OECD countries, 
17% of immigrant students scored in the top quarter 

of reading performance in the country where they sat 
the PISA test. In Brunei Darussalam, Jordan, Panama, 
Qatar, Saudi Arabia and the United Arab Emirates, 
more than 30% of immigrant students performed at that 
level. Similarly, in Australia, Brunei Darussalam, Jordan, 
Macao (China), Panama, Qatar, Saudi Arabia, 
Singapore and the United Arab Emirates, immigrant 
students scored higher than or at least at the same level 
as their native-born peers. Despite the considerable 
challenges they face, they succeed in school - a 
estament to the great drive, motivation and openness 
that they and their families possess. 


n 1954, the United States opened its borders to an 
immigrant from Syria. His son, Steve Jobs, became 
one of the world’s most creative entrepreneurs who 
revolutionised six industries: personal computers, 
film, music, telephony, tablet computing and digital 
publishing. Jobs's life story may sound like a fairy 
ale, but itis firmly rooted in reality. While immigrants 
are over-represented amongst poor performers in 
PISA, in half of PISA-participating countries and 
economies, one in five immigrant students managed 
to score in the top quarter performance in their host 
countries. Amongst countries where more than 45% 
of immigrant students are disadvantaged |including 
Austria, Denmark, Greece and Slovenia), the share 
of immigrant students who attained the top quarter 
of reading performance in their country was as large 
as the share of disadvantaged students who attained 
that level. In France and Germany, more immigrant 
students than disadvantaged students attained that 
level. These students, who manage to overcome the 
double disadvantage of poverty and an immigrant 
background, have the potential to make exceptional 
contributions to their host countries. 





Most immigrant students and their parents hold an 
ambition to succeed that in some cases surpasses 
the aspirations of families in their host country. That 
is remarkable, given that immigrant students in most 
countries are more disadvantaged and do not 
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Figure 13 ¢ Difference in reading performance, by immigrant background 
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Countries and economies are ranked in descending order of the gap in reading performance related to immigrant background, after accounting for students' socio-economic status. 


OECD average-36 refers to the arithmetic mean across OECD countries (and Colombia), excluding Spain. 


Source: OECD, PISA 2018 Database, Table 1I.B 1.9.3; Figure 11.9.6. 
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perform as well as students without an immigrant 
background. When comparing students of similar 
socio-economic status, the difference between 
immigrant and non-immigrant students in their 
expectations for their future education grows even 
larger. This is important, as students who hold ambitious 
yet realistic expectations about their future are more 
likely to put effort into their learning and make better 
use of the opportunities available to them to achieve 
their goals (Figure 14). 


The large variation in performance between immigrant 
and non-immigrant students in different countries, even 
atter accounting for socio-economic background 

and country of origin, suggests that policy can play 

a significant role in minimising those disparities. The 
key is to dismantle the barriers that usually make it 
harder for immigrant students to succeed at school. 
The crunch point is not necessarily the point of entry, 
but afterwards, when educators and school systems 
decide whether or not to offer programmes and 
support specifically designed to help immigrant 
students succeed. 


Designing education policies to address immigrant 
students’ needs — particularly language instruction — is 
not easy, and education policy alone is insufficient. 

For example, immigrant students’ performance in PISA 
is more strongly (and negatively) associated with the 
concentration of disadvantaged students in schools 
han with the concentration of immigrants or of students 
who speak a language at home that is different from 
he language of instruction. 





Reducing the concentration of disadvantage in 
schools might require changes in other social policy, 
such as housing or welfare, to encourage a more 
balanced social mix in schools. Consider this: when 

he influx of low-skilled immigrants to Europe began 

0 grow rapidly in the 1970s, the Netherlands chose 

o accommodate the migrants in large, specially 
constructed urban housing blocks. The neighbouring 
Flemish Community of Belgium, whose schools are run 
on policies very similar to those in the Netherlands, 
chose to give vouchers to migrant workers to 
supplement the amount that they would otherwise have 
to spend on housing. They could use these vouchers 
wherever they wished. The result was that there were 
fewer Flemish schools composed entirely of the sons 
and daughters of migrant workers. Years later, the 
Netherlands faced an enormous challenge to educate 
students from the public housing projects whom they 
had not been able to integrate into their education 
system and who continued to be low achievers. By 
contrast, in the Flemish Community of Belgium, where 
the migrants had been more dispersed, students 
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from immigrant families were doing far better than 
their counterparts in the Netherlands, where housing 
segregation had led to school segregation. 


Over the years, OECD's analyses have established 
some pointers for policy. A quick-win policy response 
is to provide language support for immigrant students 
with limited proficiency in the language of instruction. 
Common features of successful language-support 
programmes include sustained language training 
across all grade levels, centrally developed curricula, 
teachers who are specifically educated in second- 
language acauisition, and a focus on academic 
language, and integrating language and content 
learning. 


Since language development and general intellectual 
development are intertwined, it seems best not to 
postpone teaching the mainstream curriculum until 
students fully master their new language. What is 
important is to ensure close co-operation between 
language teachers and classroom teachers, an 
approach that is widely used in countries that seem 
most successful in educating immigrant students, such 
as Australia, Canada and Sweden. 


Offering high-quality early childhood education, 
tailored to language development, is another policy 
response. Participating in early education programmes 
can improve the chances that immigrant students start 
school at the same level as non-immigrant children. 
Targeted home visits can encourage enrolment in early 
childhood education and can help families support 
their child’s learning at home. 


However, spending on early childhood education, in 
and of itself, is not enough. Key to success is helping 
children from disadvantaged backgrounds develop the 
kinds of cognitive, social and emotional skills that they 
might not acquire at home. 


A third high-impact policy option is to build specialist 
knowledge in the schools receiving immigrant children. 
This can involve providing special education for teachers 
to better tailor instructional approaches to diverse 
student populations and support second-language 
learning. It can also help if teacher turnover is reduced 
in schools serving disadvantaged and immigrant 
populations, and if high-quality and experienced 
teachers are encouraged to work in these schools. 
Hiring more teachers from ethnic minority or immigrant 
backgrounds can help reverse the growing disparity 
between an increasingly diverse student population 
and a largely homogeneous teacher workforce, 
especially in countries where immigration is a more 
recent phenomenon. 
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Figure 14 ® Proportion of high-skilled employees in the labour force and students with realistic and ambitious expectations 
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2. WISE refers to the World Indicators of Skills for Employment; for more information, please refer to https://www.oecd.org/employment/skills-for-employment-indicators.him. 


Notes: Only countries and economies with available data are shown in this figure. 

Tertiary education corresponds to ISCED levels 5A, 5B or 6 according to the International Standard Classification of Education (ISCED- 1997). 
Countries and economies are ranked in descending order of the percentage of employed adults with tertiary education. 

OECD average-36 refers to the arithmetic mean across OECD countries (and Colombia), excluding Spain. 


Source: OECD, PISA 2018 Database, Table 1I.B 1.6.8; Figure II.6.3. 
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The harder challenge is avoiding concentrating 
immigrant students in the same, underachieving schools. 
Schools that struggle to do well for native-born students 
will struggle even more with a large population 

of children who cannot speak or understand the 
language of instruction. Countries use different 
approaches to address the concentration of immigrant 
and other disadvantaged students in particular 
schools. One way is to attract other students to 

these schools, including more advantaged students. 

A second is to better equip immigrant parents with 
information on how to select the best school for their 
child. A third is to limit the extent to which advantaged 
schools can select students. 


Extra support and guidance for immigrant parents 
can also help. While immigrant parents may have 
high aspirations for their children, they may feel 

limited in their capacity to support their children if 
they have poor language skills or an insufficient 
understanding of the school system. Programmes to 
support immigrant parents can include home visits to 
encourage these parents to participate in educational 
activities, employing specialised liaison staff to improve 
communication between schools and families, and 
reaching out to parents to involve them in school- 
based activities. 


Equity and gender 


Technically, the industrialised world had closed the 
gender gap in education — as measured in average 
years of schooling — by the 1960s. That made a huge 
difference, as about half of the economic growth in 
OECD countries over the past 50 years was due to 
higher educational attainment, mainly amongst women. 
But women still earn 15% less than men, on average in 
OECD countries, and 20% less amongst the highest-paid 
workers. Some analyses suggest that this is because 
men and women who do similar work are not paid 
the same. But a more important factor is that men and 
women pursue different careers, and those career 
choices are often made early in life. 


In mathematics and science, PISA 2018 suggests that 
gender differences are generally small (Figure 15). 
Boys outperformed girls just by five score points in 
mathematics, on average across OECD countries, 
and girls outperformed boys in science just by two 
score points. Only in Argentina, the four provinces/ 
municipalities of China, Colombia, Costa Rica and 
Mexico did boys significantly outperform girls in 
science, while the opposite was true in 33 countries 
and economies. However, while claiming victory 

in having closed gender gaps in girls’ and boys’ 
cognitive abilities, education may have lost sight of 
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other social and emotional dimensions of learning that 
may have a stronger impact on children as they think 
about what they want to be when they grow up. 


Amongst the 15-year-olds assessed by PISA, only 1% 
of girls reported that they want to work in |CT-related 
occupations, compared with 8% of boys who so 
reported, on average across OECD countries. In 
Bulgaria, Estonia, Lithuania, Poland, Serbia and 
Ukraine, more than 15% of boys reported that they 
expect to work in an ICT-related profession; but in no 
PISA-participating country or economy did more than 
3% of girls so report. It seems even when they excel in 
mathematics or science in PISA, boys and girls have 
very different expectations for their future occupation 
(Figures 16a and 16b). More than one in four boys 
reported than they expect to work as an engineer or 
science professional when they are 30 years old, but 
fewer than one in six girls so reported. Almost one in 
three high-performing girls, but only one in eight boys 
with the same proficiency reported that they expect to 
work as health professionals. 


Data from earlier PISA assessments suggest that girls 
do not seem to be getting much encouragement from 
their parents either. In all countries and economies 
surveyed on this question in 2012, parents were more 
likely to expect their sons, rather than their daughters, 
to work in a STEM field — even when boys and girls 
performed equally well in mathematics and science. 
In 2012, some 50% of parents in Chile, Hungary 

and Portugal reported that they expect their sons to 





have a career in science, technology, engineering or 
mathematics, but less than 20% of parents held such 
expectations for their daughters. 


The picture is very different when it comes to reading. 
In all countries and economies that participated in 
PISA 2018, girls significantly outperformed boys in 
reading — by 30 score points. The narrowest gender 
gaps (less than 20 score points) were observed in 
Argentina, the four provinces/municipalities of China, 
Chile, Colombia, Costa Rica, Mexico, Panama and 
Peru; the widest (more than 50 score points) were 
observed in Finland, Jordan, the Republic of North 
Macedonia, Qatar, Saudi Arabia and the United Arab 


Emirates. 


A comparison of results in reading performance 
between 2009, when reading was also the main 
subject assessed in PISA, and 2018 shows that, 

in several countries and economies, the gender 
gap in reading performance narrowed over time. 
It shrank significantly in 36 of the 62 countries and 
economies that participated in both the 2009 and 
2018 PISA assessments. In 17 of those countries 


© OECD 2019 31 


32 


PISA 2018: Insights and Interpretations 


Figure 15 * Gender gap in reading and mathematics performance 
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Note: Gender gap refers to the difference between girls and boys (girls minus boys). 
Source: OECD, PISA 2018 Database, Tables II.B 1.7.1 and 1I.B1.7.3; Figure II.7.3. 


and economies, the narrowing of the gender gap in 
reading performance was due to an improvement 

in boys' performance. However, in 11 countries, 
namely Bulgaria, Hungary, Indonesia, Italy, Japan, 
Kazakhstan, Latvia, Mexico, New Zealand, the 
Slovak Republic and Switzerland, the narrowing of the 
gender gap in reading performance was due not to an 
improvement in boys’ performance but to a decline in 
irls’ performance. 


@Q 


Students’ attitudes towards reading have changed 
over time too. In 2018, 24% of 15-year-old boys and 
44% of girls the same age agreed that “Reading is one 
of my favourite hobbies”, while 60% of boys and 39% 
of girls agreed that “| read only to get information that | 
need”. But compared with 2009 results, in 2018 larger 
proportions of both boys (an increase of 7 percentage 
points) and girls (an increase of 9 percentage points) 
agreed that “I read only if | have to”. When asked how 
much time they usually spend reading for enjoyment, 
more than 75% of boys reported either none at all 

or less than 30 minutes a day; less than 3% reported 
that they read more than two hours a day. By contrast, 
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Gender gap in reading performance (in score points) 


43% of girls reported that they read at least 30 minutes 
a day, and 8% of them reported reading more than 
2 hours a day. 


Perhaps surprisingly, the large gender gap in reading 
performance observed amongst 15-year-olds in PISA 
virtually disappears amongst the 16-29 year-olds 
who were assessed by the OECD Survey of Adult 
Skills. Why? To some extent this may have to do with 
ditferences in the cohorts; but another explanation 

is that young men are much more likely than young 
women to read at work. This suggests that there are 
many ways to narrow or even eliminate gender gaps 
in education and skills, as long as education enlists 
parents, teachers, school leaders and employers in 
giving men and women the same opportunities and 
encouragement to learn. 


The good news is that narrowing these gender gaps 
does not require expensive reform. Rather, it requires 
concerted efforts by parents, teachers and employers 
to become more aware of their own conscious or 
unconscious biases so that they give girls and boys 
equal chances for success at school and beyond. 
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Figure 16a * Gender gap in career expectations amongst top performers in mathematics and/or science 
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Notes: Statistically significant differences between girls and boys are show in a darker tone. 

For students’ career expectations, results are only available for the French Community of Belgium. 

In this figure, “top performers” refers to students who attain at least Level 2 in all three core subjects and Level 5 or 6 in mathematics and/or science. 
Countries and economies are ranked in descending order of the percentage of top performing girls who expect a career in the field. 

OECD average-36 refers to the arithmetic mean across OECD countries (and Colombia), excluding Spain. 


Source: OECD, PISA 2018 Database, Tables II.B 1.8.22 and II.B 1.8.23; Figure II.8.8. 
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Figure 16b * Gender gap in career expectations amongst top performers in mathematics and/or science 
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For students’ career expectations, results are only available for the French Community of Belgium. 

In this figure, “top performers" refers to students who attain at least Level 2 in all three core subjects and Level 5 or 6 in mathematics and/or science. 
Couniries and economies are ranked in descending order of the percentage of top performing girls who expect a career in the field. 

OECD average-36 refers to the arithmetic mean across OECD countries (and Colombia), excluding Spain. 


Source: OECD, PISA 2018 Database, Tables II.B 1.8.22 and II.B 1.8.23; Figure II.8.8. 
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For example, PISA shows clearly that boys and girls 
have different reading preferences. Girls are far 

more likely than boys to read novels and magazines 
for enjoyment while boys prefer comic books and 
newspapers. If parents and teachers were to give boys 
a greater choice in what they read, they might help 
boys develop an enjoyment of reading, which could, in 
turn, lead to at least a narrowing of the gender gap in 
reading performance. 


PISA 2015 also found that boys spend more time 
playing video games and less time doing homework 
than girls. While excessive video gaming is shown 
to be a drag on student performance, a moderate 
amount of video gaming is related to boys’ better 
performance in digital reading than in print reading 
(although boys still lag behind girls in both types of 
reading). Anyone with teenage children knows how 
ditficult it is to tell them how to spend their free time; 
but all parents should be aware that convincing their 
children that completing homework comes before 
playing video games will significantly improve their 
children's life chances. 


One of the most revealing findings from PISA 2009 
was that teachers consistently give girls better marks 

in mathematics than boys, even when boys and girls 
perform similarly on the PISA mathematics test. That 
might be because girls are “good students” — attentive 
in class and respectful of authority — while boys may 
have less self-control. But while higher marks may 
mean success at school, they are not necessarily an 
advantage for girls in the long run, particularly when 
they do not lead to higher aspirations. Labour markets 
ultimately reward people for what they know and what 
they can do with what they know, not for their grades 
at school. 


When it comes to preparing for entry into the labour 
market, PISA shows that girls are more likely than 

boys to get information about future studies or careers 
through Internet research, while boys are more likely 
than girls to get hands-on experience, by working as 
interns, job shadowing, visiting a job fair or speaking to 
career advisers outside of school (more on that later). 
This implies that employers and guidance counsellors 
can do far more to engage girls in learning about 
potential careers. 


Read more about these issues in Chapter 8 
of PISA 2018 Results (Volume II): Where All 
Students Can Succeed. 
https://doi.org/10.1787/b5fd1b8f-en 
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Figure 17 * Growth mindset, by student characteristics 


Percentage of students who disagreed or strongly disagreed with the statement: 


"Your intelligence is something about you that you can't change very much" 
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Source: OECD, PISA 2018 Database, Tables IIl.B 1.14.1 and IIl.B 1.14.3; Figure III.14.1. 
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Fostering a growth mindset 


When students struggle and teachers respond by 
lowering standards, teachers may imply that low 
achievement is the consequence of an inherent lack 

of ability. Unlike effort, talent is seen as something that 
students have no control over, so students may be 

ore likely to give up rather than try harder. According 
to some research, teachers also give more praise, 
more help and coaching, and lengthier answers to 
questions to those students whom they perceive as 
having greater ability. When teachers don't believe 
that pupils can develop and extend themselves through 
hard work, they may feel guilty pressing students whom 
they perceive to be less capable of achieving at higher 
levels. This is also concerning because research shows 
that when a teacher gives a student an easier task and 
then praises that student excessively for completing it, 
the student may interpret the teacher's behaviour as 
reflecting a belief that the student is less able. 


3 





All of this is important because of all the judgements 
people make about themselves, the most influential 

is how capable they think they are of completing a 
task successtully. More generally, research shows that 
the belief that we are responsible for the results of our 
behaviour influences motivation, such that people are 
more likely to invest effort if they believe it will lead to 
the results they are trying to achieve. 


In this context, it is worrying that in one-third of 
countries and economies that participated in PISA 
2018, more than one in two students said that 
intelligence is something about them that they can't 
change very much (Figure 17). In the Dominican 
Republic, Indonesia, Kosovo, the Republic of North 
Macedonia, Panama and the Philippines, at least 
60% of students agreed or strongly agreed with 





that statement. Those students are unlikely to make 
the investments in themselves that are necessary to 
succeed in school and in life. 


Perhaps not surprisingly then, students who disagreed 
or strongly disagreed with the statement "Your 
intelligence is something about you that you can't 
change very much" scored 32 points higher in reading 
than students who agreed or strongly agreed, after 
accounting for the socio-economic profile of students 
and schools. Students who believe that their abilities 
and intelligence can be developed over time (those 
with a “growth mindset”) also expressed less fear of 
failure than students who believe their abilities and 
intelligence are “fixed” (Figure 18). In PISA 2018, 

the students with a growth mindset reported greater 
motivation to master tasks and self-efficacy, set more 
ambitious learning goals for themselves, attached 
greater importance to school, and were more likely to 
expect to complete a university degree. 


There are various ways a growth mindset can be 
instilled in students. It can begin by teaching students 
more about the brain's capacity to learn through 
reading, class discussions and other activities. 
Research has shown that students who are exposed 
to these school-based interventions tend to show 
stronger beliefs about the brain’s capacity to change, 
and are less likely to attribute failure to a lack of 
talent, than students who are assigned to control 
groups. Other successful interventions include 
encouraging students to explain the growth mindset 
to other students, instilling a growth mindset amongst 
parents and teachers, offering a single online session 
about the growth mindset, and playing with a social 
robot that displays growth-mindset beliefs. 
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Figure 18 * Growth mindset and student attitudes 
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Source: OECD, PISA 2018 Database, Table III.B 1.14.7; Figure III.14.5. 


However, a lot comes down to the instructional 
system. In East Asia, mastery learning is often used 
to strengthen a growth mindset. It builds on the 
understanding that learning is sequential, and that 
mastery of earlier tasks is the foundation on which 
proficiency in subsequent tasks is built. According 

to this approach, student learning outcomes reflect 
the amount of time and instruction a student needs 
to learn, and whether the opportunity to learn and 
quality of instruction are sufficient to meet students’ 
needs. For teachers, that means that they do not 
vary the learning goals, which hold for the entire 
class, but that they do whatever is needed to ensure 
that each student has the opportunity to learn the 
material in ways that are appropriate to him or her. 
Some students will require additional instruction time, 
others will not; some students will require different 
learning environments than others. Behind this 
thinking is the belief that all students can learn and 
succeed, and that the task of teachers is to design the 
learning environments, whether inside or outside the 
assroom, that help students realise their potential. 
Because all students succeed in completing each 
successive task, the result is often less variation and a 
weaker impact of socio-economic status on learning 
outcomes. 





oO 


However, regardless of whether or not students believe 
that they can develop their intelligence, students may 
find it challenging to do so if they are not given the 
necessary tools and support. Parents, teachers and 
principals need fo create an environment where 
children are encouraged to participate, and where 
educators believe in students’ potential to develop their 
skills and provide students with the necessary support 
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and feedback. In this regard, PISA 2018 results reveal 
that students who perceived their teachers to be more 
supportive scored higher in reading, particularly after 
accounting for their socio-economic status. Policy 
makers, in turn, need to provide educators with the 
necessary resources and time to achieve their goals. 
When the role played by educators is not recognised 
as essential for encouraging a growth mindset to 
take root and flourish, the responsibility for failing lies 
entirely with the student, even when they do not have 
the resources necessary to reach their full potential. 


Finland's special teachers work closely with classroom 
teachers to identity students in need of extra help, and 
then work individually or in small groups with struggling 
students to help them keep up with their classmates. 

It is not left solely to the regular classroom teacher to 
identify a problem and alert the special teacher; every 
comprehensive school has a “pupils’ multiprofessional 
care group” that meets at least twice a month for two 
hours. The group consists of the principal, the special 
teacher, the school nurse, the school psychologist, 

a social worker, and the teachers whose students 

are being discussed. The parents of any child being 
discussed are contacted prior to the meeting and are 
sometimes asked fo attend. 


In many countries, it has taken time to move from 

a belief that only a few students can succeed to 
embracing the idea that all students can achieve 

at high levels. It takes a concerted, multifaceted 
programme of policy making and capacity building 
to attain that goal. But one of the patterns observed 
amongst the highest-performing countries is the 
gradual move from a system in which students were 


stratified into different types of secondary schools, with 
curricula demanding various levels of cognitive skills, to 
a system in which all students go to secondary schools 
with similarly demanding curricula. 


Amongst OECD countries, Finland was the first to 

take this route in the 1970s; Poland was the most 
recent, with its school reform in the 2000s. These 
countries “levelled-up", requiring all students to meet 
the standards that they previously expected only their 
elite students to meet. Students who start to fall behind 
are identified quickly, their problem is promptly and 
accurately diagnosed, and the appropriate course of 
action is quickly taken. Inevitably, this means that some 
students are targeted for more resources than others; 
but it is the students with the greatest needs who benefit 
from the most resources. 


It takes strong leadership, and thoughtful and sustained 
communication to bring parents along in this effort, 
particularly those benefiting from the more selective 
tracks. In the end, education systems are unlikely to 
sustain high performance and equitable opportunities 
to learn without the premise that it is possible for 

all students to achieve at high levels — and that it is 
necessary for them to do so. 


Students’ confidence in their abilities and their fear 

of failure also affect both their performance and their 
well-being. While a moderate sense of fear can spur 
students to expend greater effort on academic tasks, 
an excess of fear could compel students to avoid 
challenging tasks and situations that are essential for 
their personal growth. PISA found that, on average, the 
greater the fear of failure expressed by students, the 
higher a country’s reading score. This was observed 
in a large number of English-speaking and East Asian 
education systems. 


However, fear of failure can be a double-edged 
sword, and parents, teachers and school principals 
should be aware that instilling a fear of failure in 
children may also adversely affect their well-being. For 
instance, PISA results show that, in virtually all school 
systems, students reported less satisfaction with life 
when they expressed a greater fear of failure. Policy 
makers may be interested in learning more about the 
Flemish Community of Belgium, Estonia, Finland and 
Germany — education systems where students scored 
above the OECD average in reading, but expressed 
less fear of failure than the typical student across 
OECD countries. Interestingly, PISA also found that, 

in every school system except the Flemish Community 
of Belgium and Germany, students who exhibited a 
growth-mindset — i.e. they believe that abilities and 
intelligence can be developed over time — reported 
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less fear of failure than students with a fixed mindset. 


PISA results also show that students who see 
themselves as more competitive scored higher in 
reading than those who do not, especially when 
they reported trying harder when in competition 

with others, and this holds for both boys and girls. 
However, parents, teachers, school principals and 
policy makers should be aware that when they 
create a competitive learning environment, not all 
students may respond in the same way. For example, 
PISA results show that in a competitive environment, 
boys and students who perceived themselves as 
competitive reported better well-being outcomes than 
girls and students who did not perceive themselves 
as competitive. In the debate about how much 
competition and co-operation are needed, some 
researchers argue that when co-operative and 
competitive behaviours are intertwined, as in inter- 
team competitions, the performance and enjoyment 
of participants are even higher than in a purely co- 
Operative or competitive environment. 


Read more about these issues in Chapters 12, 
13 and 14 in PISA 2018 Results (Volume III): 
What School Life Means for Students’ Lives. 
https://doi.org/10.1787/acd78851-en 
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Aligning education and career aspirations 


Across the world, young people who leave education 
today are entering the labour market with often 
considerably more years of schooling than their 
parents or grandparents had when they started 
working. And yet, young people continue to struggle 
in the labour market, employers continue to complain 
that they cannot find the new talent they need, and 
governments continue to worry about the mismatch 
between what the labour market demands and 

what the education system supplies. The bottom line 
is that, for young people, academic success alone 

is not sufficient to ensure easy transitions into good 
employment. 


The PISA 2018 assessment collected data not only 

on the knowledge and skills of 15-year-olds, but 

also on their expectations for further education and a 
career (Table 1). These data allow for an examination 
of how young people’s career aspirations compare 
with future-focused labour market information about 
specific occupations — their potential for growth and 
their likelihood of succumbing to automation. It is also 
possible to examine how closely young people's 
career aspirations correlate with their strengths and 
skills, and to identity gaps where additional awareness 
and support could help young people discover, pursue 
and secure careers in which they might be successful. 


These data also make it possible to examine whether 
gender-related differences in career expectations 
have already taken root by the age of 15. Location 
data can also be leveraged to analyse differences 
in ambition across countries and between urban 

and rural environments. They can also be used to 
identify regional factors to be borne in mind when 





giving young people information about labour-market 
opportunities. The PISA 2018 report includes an initial 
analysis on this, and more detailed analyses are 
underway. 


Of course, not all 15-year-olds have a clear vision 

of their future career, and some 25% of students, 

on average across OECD countries, provided only 
vague answers fo the question about their career 
expectations. However, longitudinal studies in four 
countries that followed students who had sat the PISA 
test in the early 2000s as they became young adults 
found that those who, at the age of 15, had expected 
to work in a high-skilled job were more likely to be 
doing so as young adults than those who had not held 
high expectations when they were 15. 


The results from PISA 2018 show that disadvantaged 
students tend to hold lower ambitions than would be 
expected given their academic achievement (Figure 19). 
More than nine in ten high-achieving advantaged 
students, but only seven in ten high-achieving 
disadvantaged students reported that they expect to 
complete tertiary education. As a result, more than 
30% of high-achieving disadvantaged students did 
not expect to complete tertiary education; and even 
when disadvantaged students aspired to high-skilled 
occupations, a considerable proportion of them held 
expectations of future education that were not aligned 
with their career goals. 


Encouraging all students, especially disadvantaged 
students, to have ambitious and realistic education and 
career expectations is necessary not only to improve 
equity but as an investment in the future. Only in a few 
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Figure 19 _* High performers who do not expect to complete tertiary education, by socio-economic status 





Percentage of students amongst those who have attained at least minimum proficiency (Level 2) in the three core PISA subjects and are high 
performers (Level 4) in at least one subject 
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Notes: The percentage of high performers is shown next to the country/economy name. 

Statistically significant differences are marked in a darker tone. 

Only countries and economies with sufficient proportions of high performers amongst advantaged/disadvantaged students are shown in this figure. 
Countries and economies are ranked in ascending order of the percentage of advantaged students. 

OECD average-36 refers to the arithmetic mean across OECD countries (and Colombia), excluding Spain. 


Source: OECD, PISA 2018 Database, Table II.B 1.6.7; Figure II.6.5. 
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Table 1 * Top 10 career expectations of 15-year-old students, by gender 





























Ist Police officers Specialist medical practitioners 
2nd Athletes and sports players Generalist medical practitioners 
3rd Engineering professionals Lawyers 

Ath Generalist medical practitioners Teaching professionals 

5th Business services and administration managers Nursing professionals 

6th otor vehicle mechanics and repairers Medical doctors 

7th Armed forces occupations, other ranks Psychologists 

8th Policy and planning managers Police officers 

Oth Lawyers Veterinarians 

10th Teaching professionals Policy and planning managers 








Source: OECD, PISA 2018 Database; Table 11.6.1. 


countries, such as Canada, Chile, Korea, Singapore, 
Ukraine and the United States, were students’ 
expectations of further education both ambitious and 
aligned with their academic performance, regardless 
of the students’ socio-economic status. 


In addition, disadvantaged students are often more 
at risk of lacking relevant information about future 
education and career choices. In many countries, 
schools that enrol more disadvantaged students were 
less likely to provide opportunities for students to 
discuss their career plans with a specialised advisor, 
on average. In fact, PISA 2018 finds that only a 
small proportion of disadvantaged students reported 
knowing how to get information about student 
financing (e.g. student loans or grants) for higher 
education. 


A misalignment between academic performance, on 
the one hand, and education and career expectations, 
on the other, can be partly due to the anticipated 
ditficulties in progressing through a long and costly 
education. For example, an experiment conducted in 
the Dominican Republic suggests that eighth-grade 
boys from poor backgrounds largely underestimated 
the returns to higher education, and that providing them 
with accurate information had a positive impact on 
their schooling. 


A study in the United States showed the potential 
of intensive college counselling provided to 
college-aspiring, low-income students. These 
interventions are typically run by community-based 
non-profit organisations, and provide personalised 
guidance to students throughout the college search, 
application and financial aid processes. They shift the 
focus towards enrolment in four-year colleges that are 








less expensive and have higher graduation rates than 
the alternatives that students would otherwise choose. 
Counselling also improves students’ persistence 
through at least the second year of college, suggesting 
a potential to increase the rate of degree completion 
amongst disadvantaged students. Similar results were 
observed with another intervention, which showed 
that mailing high-achieving seniors an information 
packet and application fee waivers made low-income 
students more likely to enrol in colleges that have 
stronger academic records and higher graduation 
rates than those to which students with similar profiles 
would normally have applied. 


An experiment in disadvantaged high schools in 
Toronto, Canada, found that watching a video about 
the benefits of post-secondary education and being 
invited to try out a financial-aid calculator significantly 
assuaged the concems of disadvantaged high school 
students about the costs of higher education, and 
raised their expectations to earn a degree. Results from 
another controlled trial conducted in German high 
schools suggest that similar low-cost interventions may 
eventually lead to greater tertiary enrolment amongst 
students whose parents did not attain that level of 
education. Students in selected schools who had 
attended a simple, in-class presentation on the benefits 
and costs of higher education, and on possible funding 
options, more often applied fo university and were 
more often enrolled than students who had not been 
exposed to these interventions. 


Similarly, in several countries, schools that enrol 
mostly disadvantaged students were less likely than 
schools that mostly enrol advantaged students to 
provide opportunities for students to discuss their 
career plans with a specialised advisor (Figure 20). 
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Figure 20 ¢ Advantaged/disadvantaged schools where one or more dedicated counsellor(s) provide career guidance 
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Notes: Statistically significant differences are marked in a darker tone. 
For this analysis, the sample is restricted to schools with the modal ISCED level for 15-year-old students. 
Couniries and economies are ranked in descending order of the percentage of studenis in advantaged schools. 


Source: OECD, PISA 2018 Database, Table 1I.B 1.6.9; Figure 11.6.6. 
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The Mexican anti-poverty programme, PROGRESA, 
shows that simply being exposed to highly educated 
professionals, such as doctors and nurses, raises 

the aspirations of poor families for their children’s 
education, and has a positive impact on students’ 
achievement at school. 


Clearly, school-to-work transitions have become more 
complex and prolonged. This reflects both changes 
in labour market demand, and a growing disparity 
with what education systems supply. This generation 
of young citizens requires curiosity, entrepreneurship 
and resilience to work effectively in the new labour 
market. They will need confidence to create their 
own employment and to manage their careers in 
new ways. Education systems need to prepare young 
people to be effective in applying what they know 

to ever-changing situations, expose them to relevant 
role models, and provide guidance that helps them 
discover their passions, areas where they can excel, 
and where and how they can find or create a job. 


Read more about these issues in Chapter 6 
in PISA 2018 Results (Volume II): Where All 
Students Can Succeed. 
https://doi.org/10.1787/b5fd1b8f-en 
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Figure 21 ¢ Students’ well-being, by academic resilience 
Percentage-point difference between students who are academically resilient and those who are not 
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Notes: Statistically significant differences between students who are resilient and those who are not are shown in a darker tone. 

Resilient students are disadvantaged students who score in the top quarter of performance in reading amongst students in their own country. 

Non-resilient students are disadvantaged students who do not score in the top quarter of performance in reading. 

Students with positive well-being refers to students who reported that they are satisfied with their lives, do not feel like outsiders at school and do not doubt their future plans when facing failure. 
For the index do not doubt their future plans when faced with failure, data are only available for the Flemish Community of Belgium. 

Countries and economies are ranked in descending order of the percentage-point difference between students who are academically resilient and those who are not. 


Source: OECD, PISA 2018 Database, Table 1I.B 1.3.5; Figure II.3.8. 
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School life, student lite 


From its first assessment, PISA has been asking students 
about their motivations and dispositions towards 
learning, such as their enjoyment of reading and 

their anxiety towards mathematics. In more recent 
assessments, PISA also asked students about their 
more general social and emotional state, including 
their satisfaction with life, their feelings and their 

fear of failure, in order to establish a more holistic 
appreciation of education outcomes and student 
well-being. In addition, an optional questionnaire on 
well-being was distributed as part of PISA 2018. All 

of these questions connect school life with the broader 
ecosystem in which students live — the family, their 
peers, the community — and provide information on the 
development of 15-year-old students. 


Across OECD countries, about two in three students 


reported that they are satis 
percentage that shrank by 
between 2015 and 2018. 


fied with their lives, a 
five percentage points 


More than 85% of students 


reported sometimes or always feeling happy, cheerful 


or joyful; but about 6% of s 





udents reported always 


feeling sad. In almost every education system, girls 
expressed greater fear of failure than boys, even when 
they outperformed boys in reading by a large margin, 
and this gender gap was considerably wider amongst 
top-performing students. Positive student well-being 
was also associated with a higher proportion of 
resilient students (Figure 21). 


PISA 2018 shows that school life is closely related to 
the well-being of 15-year-old students. For instance, 
the three aspects of students’ lives that are more 
strongly associated with expressions of sadness are 
how satisfied students are with the way they look, with 
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and student well-being 


their relationships with their parents, and with school 
life. Ultimately, the well-being of students may also 
atfect their academic performance. In this regard, PISA 
data show that students who were frequently bullied 
were more likely to have skipped school and scored 
lower in reading. 


A positive school climate is one of those things that 

is difficult to define and measure, but everyone 
recognises it when they see it. Students appreciate 

a school environment where bullying is unusual, 
where students do not feel out of place, and where 
establishing genuine and respecttul relationships with 
teachers is the norm. PISA 2018 shows that school 
climate is closely associated with students’ sense of 
well-being. 


The disciplinary climate in language-of-instruction 
lessons is also one of the strongest predictors of 
reading performance. In all countries and economies, 
students who reported fewer disciplinary problems in 
their language-of-instruction lessons performed better 
in reading, after accounting for the socio-economic 
profile of students and schools. More specifically, 

on average across OECD countries, students who 
reported that students cannot work well in every or 
most language-of-instruction lessons scored 25 points 
lower in reading than students who reported that this 
never happened or happened only in some lessons, 
after accounting for socio-economic status. Even 
occasional disciplinary problems were negatively 
associated with reading performance. Students who 
reported that disciplinary problems occur in some 
language-of-instruction lessons scored between 

5 and 9 points lower in reading than students who 
reported that the problems never, or hardly ever, occur. 
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Figure 22 ¢ Prevalence of teacher enthusiasm and reading performance 





OECD average; reference category "strongly disagree" 
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Results based on linear regression analysis, after accounting for students’ socio-economic profile, gender and immigrant background. The socio-economic profile is measured by the PISA 


index of economic, social and cultural status. 


Source: OECD, PISA 2018 Database, Table IIIB 1.5.7; Figure III.5.3. 


The good news is that the disciplinary climate in 
school generally improved between 2009 and 2018, 
according to students’ reports, especially in Albania, 
Korea and the United Arab Emirates. For example, 
on average across OECD countries, the percentage 
of students who reported that their classmates in their 
language-of-instruction lessons always, or almost 
always, listen to what the teacher says, or can work 
well, increased by about four percentage points 
during that period. That's good news because all 
types of students appeared to benefit from a positive 
disciplinary climate. The relationship between 
disciplinary climate and reading performance was 
relatively stable across students’ gender, socio- 
economic status and immigrant background. 


PISA 2018 also asked students whether their 
language-of-instruction teacher supports them in 

their schoolwork and is enthusiastic about teaching. 
Around three in four students reported that, in most 

or every lesson, the teacher gives extra help when 
students need it and that the teacher helps students 
with their learning. In most countries and economies, 
students scored higher in reading when they perceived 
their teachers as more enthusiastic, especially when 
they said their teachers were interested in the subject 
(Figure 22). On average across OECD countries and 
in 43 education systems, students who perceived 
greater support from teachers scored higher in reading, 
after accounting for the socio-economic profile of 
students and schools. Equally important, teacher 
enthusiasm and teachers’ stimulation of reading 
engagement were the teaching practices most strongly 
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(and positively) associated with students’ enjoyment of 
reading (Figure 23). 


Of course, most teachers care about having positive 
relationships with their students, but some teachers 
might be insufficiently prepared to deal with difficult 
students and classroom environments. Effective 
classroom management consists of far more than 
establishing and imposing rules, rewards and incentives 
to control behaviour; it requires the ability to create 

a learning environment that facilitates and supports 
students’ active engagement in learning, encourages 
co-operation, and promotes behaviour that benefits 
other people. A stronger focus on classroom and 
relationship management in professional-development 
programmes may give teachers some of the tools they 
need to connect better with their students. Teachers 
also need the time to share information about students’ 
strengths and weaknesses with their colleagues, so 
that, together, they can find the best approaches to 
make students feel part of the school community. 


Clearly, what goes on at school can have an impact 
not only on students’ attitudes towards learning, but on 
their feelings, in general (Figure 24). Parents recognise 
this, too, as they cited school safety, school climate and 
school reputation as the most important criteria when 
choosing a school for their child, followed closely by 
students’ academic achievement and the offering of 
specific subjects or courses. PISA 2018 finds that in 

all 65 countries and economies with available data, 
students were more likely to express positive feelings, 
in general, when they reported a stronger sense of 
belonging at school; and in virtually all school systems, 


students who perceived their peers to be more co- 
operative were more likely to express positive feelings. 


Teachers and principals often count on parents to help 
them create a positive learning environment in their 
schools. According to school principals, about 41% 
of students’ parents discussed their child’s progress 
with a teacher on their own initiative and 57% did so 
on the initiative of teachers. PISA finds that parents’ 
involvement in their child's education is positively 
associated with student performance. The average 
score in reading was higher in those countries and 
economies where more parents discussed their 
child’s progress on the initiative of teachers, and that 
positive association remained even after accounting 
for per capita GDP and for other forms of parental 
involvement in school-related activities. In fact, for 
every 10 percentage-point increase in the share 

of parents who discussed their child's progress on 
the teachers’ initiative, the average reading score in 
the country or economy improved by 10 points on 
average across the 74 countries and economies with 
available data, after acconting for national income 
and other factors. The prevalence of parents discussing 
their child's progress on the initiative of teachers may 
be an indication of a school system's responsiveness. 


At the same time, PISA 2018 also finds that fewer than 
one in five parents was involved in school government 
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or volunteered to participate in extracurricular activities 
in their child's school. Why were so few parents 
involved in these school-based activities? On average 
across the nine OECD countries and economies 

that distributed the parent questionnaire, the issues 

that parents most commonly cited as hindering their 
participation in these activities were time-related, 

and included the need to work (34%) and the 
inconvenience of meeting times (33%). Only 5% of 
parents reported that they felt unwelcome at school. 
Given these results, school leaders and educators can 
perhaps do more to accommodate working parents 
so that everyone — schools, teachers, students and the 
parents themselves — can benefit from greater parental 
involvement in school activities. 


Allin all, one way to promote students’ well-being is 
to encourage all parents to be more aware of their 
children’s interests and concerns, and show interest in 
their school life, including in the challenges children 
face at school. Schools can create an environment of 
co-operation with parents and communities. Teachers 
can be given better fools to enlist parents’ support, 
and schools can address some critical deficiencies 
amongst disadvantaged children, such as the lack of a 
quiet space for studying. If parents and teachers were 
to establish relationships based on trust, schools could 
rely on parents as valuable partners in the education of 
their students. 


Figure 23 * Enjoyment of reading and teaching practices in language -of-instruction lessons 
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Results based on linear regression analysis, after accounting for students’ and schools’ socio-economic profile. The socio-economic profile is measured by the PISA index of econo- 
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Source: OECD, PISA 2018 Database, Table III.B 1.6.10; Figure III.6.5. 
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Figure 24 ¢ Predictors of positive feelings 
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Source: OECD, PISA 2018 Database, Table IIB 1.12.19; Figure III.12.5. 


50 © OECD 2019 





heerful”. 


Yet school principals and teachers may be 
overlooking some challenges if they only pay 
attention to what is happening inside the classroom. 
Although a majority of students reported that they 
feel they belong at school - across OECD countries, 
about 7 in 10 students agreed or strongly agreed that 
they feel like they belong at school — students’ sense of 
belonging at school weakened considerably between 
2003 and 2015 and waned even further between 
2015 and 2018. Even students in Japan and Korea, 
who enjoyed one of the best disciplinary climates of all 
PISA-participating countries — e.g. they rarely skipped 
school or arrived late for school, and a clear majority 
of them reported that they had never been bullied — 
were some of the most dissatisfied with their lives, at 
least according to their own reports. In addition, they 
expressed greater fear of failure, and were about 
twice as likely as students in other OECD countries to 
report that they always feel scared or sad. 





Furthermore, the share of students who reported 

being frequently bullied increased by around four 
percentage points since 2015, on average across 
OECD countries. More than one in five students 
reported being bullied at school at least a few times a 
month (Figure 25). Most of the reported bullying was 
verbal or relational — others made fun of the student, 
the student was the object of nasty rumours, or the 
student was left out of things on purpose — rather than 
physical. For example, more than 10% of students in 67 
out of 75 countries and economies reported that their 
peers made fun of them at least a few times a month; 
but on average across OECD countries, around 7% of 
students reported that they got hit or pushed around by 
other students that often. 


> 


Bullying can have adverse — and potentially long- 
asting — effects on students’ performance at school 
and general well-being. Students who reported being 
requently bullied scored 21 points lower in reading 
han students who did not report so, after accounting 
or socio-economic status. Frequently bullied students 
reported feeling sad, scared and less satisfied with 
heir lives. These students were also more likely to have 
skipped school in the two weeks prior to the PISA 

est — an indication that they missed out on valuable 
earning opportunities. 








Yet when asked about their feelings towards bullying, 
students overwhelmingly reported negative attitudes 
towards bullying — and positive attitudes towards 
defending the victims of bullying (Figure 26). For 
example, on average across OECD countries, 90% 

of students agreed or strongly agreed that they like it 
when someone stands up for other students who are 
being bullied; and 88% agreed or strongly agreed that 
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it is a good thing to help students who can't defend 
themselves. Policy makers and local educators can 
capitalise on these sentiments to put in place measures 
and programmes to combat and prevent bullying. 


Co-operation amongst students, independent of good 
relations with teachers, is also associated with higher 
performance — and with students’ well-being: in every 
participating school system, students were more likely 
to feel they belong at school when their peers were 
more co-operative. 


Given the harmful consequences that bullying can 
have on students’ well-being, policy makers, principals 
and teachers need to devise effective policies and 
practices to limit bullying. Previous OECD studies 
suggest, in particular, that: 


» 


» 


» 


» 


» 


» 


Early signs of bullying should not be overlooked. 


All types of bullying need to be taken seriously, 
including the less “visible” ones, such as relational 
orms of bullying. Considering all types of bullying 
may draw greater attention to the bullying most 
ypically suffered by girls. While boys are more 
ikely than girls to be frequently — and physically - 
bullied, the gender gap almost disappears when 
students were asked about relational types of 
bullying, such as spreading nasty rumours and being 
eft out of things on purpose. 





Monitoring students’ attitudes towards bullying 
can provide valuable insights into how to address 
bullying. For instance, students’ attitudes can be 
used as a predictive tool, to understand the role 
played by bystanders, or to identity students 

who would stand up against bullies. Changing 
bystanders’ reactions to bullying may be an 
effective way to reduce the incidence of bullying. 


Building a culture of good behaviour, establishing 
clear anti-bullying rules, and creating a positive 
school climate, where students feel engaged and 
socially connected, is essential for preventing 
bullying. 


Students and teachers should be taught how to 
recognise and respond to bullying. 


Communication with the parents of the bullied 
students and the bully him/herself is important. 


Compared to the average student across OECD 
countries, students in Spain reported being bullied 
less frequently, were more satisfied with their lives, 
expressed more positive and fewer negative feelings, 
and their sense of belonging at school was amongst 
the strongest across all PISA-participating school 


systems. Yet, according to students’ reports, the 





disciplinary climate in language-of-instruction lessons 
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Figure 25 * Being bullied, by student characteristics 
Based on students’ reports 


| Positive difference [| Negative difference 














Difference is not significant | Missing values 


A | Girls - boys Bi Advantaged - disadvantaged students ISCED 3 - ISCED 2 | D | Immigrant - non-immigrant students B High - low-achieving students in reading 


52 















































































































































































































































© OECD 2019 


0 10 20 30 40 50 60 70% 









































Note: Low-achieving (high-achieving) students are students who score amongst the bottom 25% (the top 25%) of students within their country or economy on the PISA test. 
Countries and economies are ranked in descending order of the percentage of students being bullied at least a few times a month. 


Source: OECD, PISA 2018 Database, Tables II.B1.2.1 and IIIB 1.2.4; Figure III.2.3. 
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Figure 26 * Students’ attitudes towards bullying, by gender 
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Source: OECD, PISA 2018 Database, Tables III.B1.2.15 and III.B 1.2.16; Figure III.2.5. 
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was far from ideal, and the share of students who had 
skipped school or lessons in the two weeks prior to 
the PISA test was clearly above the OECD average. A 
similar picture emerged in Costa Rica and Portugal. 


In Sweden, students’ reports on bullying, the 
disciplinary climate and student truancy were more 
positive than the reports of the average student across 
OECD countries. Moreover, students in Sweden were 
as satisfied with lite and expressed a similar sense of 
belonging at school and fear of failure as the average 
student across OECD countries. However, students in 
Sweden were more frequently late for school and were 
somewhat less likely to express positive feelings than 
students in other OECD countries. 


Read more on these issues in PISA 2018 
Results (Volume III): What School Life Means for 
Students’ Lives. 
https://doi.org/10.1787/acd78851-en 
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What comes nexte 


The three volumes of PISA 2018 results that are 
published in 2019 and summarised in this brochure 
provide the first findings from this latest assessment. 
Volume V, which will be published in June 2020, will 
highlight some of the policies and practices that predict 
the success of students, schools and education systems; 
but it will take some time until we fully understand 

the results from PISA 2018. Policy makers’ hunger for 
immediate answers is always frustrated by the snail's 
pace at which the development of data, evidence and 
research advances; and the data collected by PISA 
alone leave many questions unanswered. The results 
offer a snapshot of education systems at a certain 
moment in time; but they do not - they cannot — show 
how the school systems got to that point, or the 
institutions and organisations that might have helped or 
hindered progress. In addition, the data do not really 
say much about cause and effect. Correlations are 
often deceptive: if the birds sing when the sun rises, 
and they do so day after day, year after year, and 

in many different places around the world, it doesn't 
mean the sun rises because the birds sing. In a nutshell, 
knowing what successful systems are doing does not 
yet tell us how to improve less-successful systems. That 
is where the OECD brings a range of other tools to 
bear to strengthen insights for policy and practice. 


PISA is not only the world’s most comprehensive 

and reliable international comparison of students’ 
capabilities, it is also integrated with a range of 
methods and resources at the OECD, including country 
and thematic policy reviews, that countries can use 

to situate the results from PISA in the different contexts 
in which students learn, teachers teach and schools 





operate, and fo fine-tune their education policies. 
Education policy makers and practitioners can benefit 
from these tools in the same way that business leaders 
learn to steer their companies towards success: by 
taking inspiration from others, and then adapting 
lessons learned to their own situation. 


Sharing insights, across borders, to improve quality, 
equity and efficiency in education is more urgently 
needed than ever before. The demands on education 
and education policy are high and rising. In the past 

it was sufficient for education to sort students because 
our economies and societies could rely on a few highly 
educated individuals. In today's world, everyone 
needs to have advanced knowledge and skills, not just 
for economic reasons but also for social participation. 


In traditional bureaucratic school systems, teachers 

are left alone in classrooms with a lot of prescriptions 
about what to teach. The OECD Teaching and Learning 
International Survey, TALIS, shows the value of teachers 
and schools looking outward to collaborate with the 
next teacher and the next school. The past was about 
delivered wisdom; the future is about user-generated 
wisdom. The past was about divisions: education 
systems could allow for teachers and content to be 
divided by subjects and student abilities. The past 
was about isolation: schools were designed to keep 
students inside, and the rest of the world outside. 

The past was about hierarchies: teachers provided, 
students received. The future of education needs to be 
about integration: the integration of different subjects, 
the integration of diverse students and the integration 
of various learning contexts; it needs to be about 
connections: connections with real-world contexts, 
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and with the rich array of resources in the community; 
and it needs to be about co-creation: recognising both 
students and adults as resources for how learning is 
designed and how students succeed. 


In the past, different students were taught in similar 
ways; now, schools need to embrace diversity with 
differentiated pedagogical practices. The goals of the 
past were standardisation and compliance: students of 
the same age were educated in batches and followed 
the same instruction. The future is about personalising 
educational experiences: building instruction out of 
students’ passions and capabilities, helping students 
with individualised learning and assessment in ways 
that foster their engagement and talents. In the past, 
schools were technological islands, where technology 
was deployed mostly to support existing practices. 
The schools of tomorrow will use the potential of 
technologies to liberate learning from conventions and 
connect learners in new and powerful ways. 


The future is about more innovative partnerships. 
Effective learning environments are constantly 
creating synergies and finding new ways to enhance 
professional, social and culivral capital with others. 
They do that with families and communities, with 
higher education, with other schools and learning 
environments, and with businesses. Education needs 
to find better ways to recognise, reward and give 
exposure to the successes of innovators. 


All of this has profound implications for schools, 
teachers and teaching. The past was about 
prescription, the future requires a knowledge-rich 
profession, and the replacement of the industrial 

work organisation, with its administrative control, with 
much more professional and collaborative working 
norms. When teachers feel a sense of ownership 
over their classrooms, when students feel a sense of 
ownership over their learning, that is when learning 
for the posttruth information age can take place. 

The central reason why teachers’ ownership of the 
profession is a must-have rather than an optional 
extra is the pace of change. Even the most efficient 
attempts to push a central curriculum into classroom 
practice will drag out over a decade, because it takes 
so much time to communicate the goals and methods 
through the different layers of the system. In this age of 
accelerations, such a slow process inevitably leads to 
a widening gap between what students need to learn 
and what teachers teach. When fast gets really fast, 
being slow to adapt makes us really slow 


While measurement is the means, the purpose of PISA 
is to help countries look outward and incorporate 
the results of that learning into policy and practice. 
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That outward-looking perspective also seems to be 

a common trait of many high-performing education 
systems: they are open to the world and ready to learn 
from and with the world’s educatian leaders; they do 
not fee! threatened by alternative ways of thinking. 


in the end, the laws of physics apply. If we stop 
pedalling, not only will we not move forward, our 
bicycles will stop moving at all and will fall over - and 
we will fall with them. Against strong headwinds, we 
need to push ourselves even harder. In the face of 
challenges and opportunities as great as any that have 
gone before, human beings need not be passive or 
inert. We have agency, the ability to anticipate and 
the power fo frame our actions with purpose. The 
best-performing PISA countries show that high-quality 
and equilable education is an attainable goal, that it 
is within countries’ means to deliver a future for millions 
of learners who currently do not have one. The task is 
not to make the impossible possible, but to moke the 
possible attainable. 
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PISA 2018 Results 


Volume I, What Students Know and Can Do, provides a detailed examination of student 
performance in reading, mathematics and science, and describes how performance has changed 
since previous PISA assessments. 


https://doi.org/10.1787/5f07c754-en 


Volume II, Where All Students Can Succeed, examines gender differences in student performance, 
and the links between students’ socio-economic status and immigrant background, on the one hand, 
and student performance and well-being, on the other. 


https://doi.org/10.1787/b5td1b8f-en 





Volume III, What School Life Means for Students’ Lives, focuses on the physical and emotional 
health of students, the role of teachers and parents in shaping the school climate, and the social life at 
school. The volume also examines indicators of student well-being, and how these are related to the 
school climate. 


https://doi.org/10.1787/acd78851-en 


Volume IV, Are Students Smart about Money?, examines 15-year-old students’ understanding 


about money matters in the 21 countries and economies that participated in this optional assessment, 
forthcoming in 2020. 


Volume V, Effective Policies, Successful Schools, analyses the policies and practices used 


in schools and school systems, and their relationship with education outcomes more generally; 
forthcoming in 2020. 


Volume VI, Are Students Ready to Thrive in Global Societies?, explores students’ ability to 
examine local, global and intercultural issues, understand and appreciate different perspectives and 
world views, interact respecttully with others, and take responsible action towards sustainability and 
collective well-being; forthcoming in 2020. 
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